[
https://issues.apache.org/jira/browse/IGNITE-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347989#comment-16347989
]
Nikolay Izhikov commented on IGNITE-7337:
-----------------------------------------
{{OPTION_WRITE_PARTITIONS_NUM}} - Is number of partition we want to use for
DataFrame writing.
Each partition will be written concurrently by separate Spark worker.
If {{OPTION_WRITE_PARTITIONS_NUM}} is greater then DataFrame already has -
DataFrame will remaing the same.
Actulally, I take this option from Spark internal JDBC implementation.
* Usage in my PR -
https://github.com/apache/ignite/pull/3438/files#diff-69b49192927a69d21b4afec604f60b77R105
* coalesce doc -
https://spark.apache.org/docs/2.2.0/api/scala/index.html#org.apache.spark.sql.Dataset@coalesce(numPartitions:Int):org.apache.spark.sql.Dataset[T]
* usage inside Spark JDBC implementation -
https://github.com/apache/spark/blob/v2.2.0/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L775
Do you think such option will be usefull for Ignite?
> Spark Data Frames: support saving a data frame in Ignite
> --------------------------------------------------------
>
> Key: IGNITE-7337
> URL: https://issues.apache.org/jira/browse/IGNITE-7337
> Project: Ignite
> Issue Type: New Feature
> Components: spark
> Affects Versions: 2.3
> Reporter: Valentin Kulichenko
> Assignee: Nikolay Izhikov
> Priority: Major
> Fix For: 2.5
>
>
> Once Ignite data source for Spark is implemented, we need to add an ability
> to store a data frame in Ignite. Most likely if should be enough to provide
> implementation for the following traits:
> * {{InsertableRelation}}
> * {{CreatableRelationProvider}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
