[jira] [Commented] (SPARK-16593) Provide a pre-fetch mechanism to accelerate shuffle stage.
[ https://issues.apache.org/jira/browse/SPARK-16593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431070#comment-15431070 ] Biao Ma commented on SPARK-16593: - I had made new commits. > Provide a pre-fetch mechanism to accelerate shuffle stage. > -- > > Key: SPARK-16593 > URL: https://issues.apache.org/jira/browse/SPARK-16593 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Biao Ma >Priority: Minor > Labels: features > > Currently, the `NettyBlockRpcServer` will reading data through BlockManager, > while the block is not cached in memory, the data should be read from DISK > first, then into MEM. I wonder if we implement a mechanism add a message > contains the blockIds that the same as the openBlock message but one loop > ahead, then the `NettyBlockRpcServer ` will load the block ready for transfer > to the reduce side. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16593) Provide a pre-fetch mechanism to accelerate shuffle stage.
[ https://issues.apache.org/jira/browse/SPARK-16593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387400#comment-15387400 ] Biao Ma commented on SPARK-16593: - There is some error in this PR, I'll create another later. > Provide a pre-fetch mechanism to accelerate shuffle stage. > -- > > Key: SPARK-16593 > URL: https://issues.apache.org/jira/browse/SPARK-16593 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Biao Ma >Priority: Minor > Labels: features > > Currently, the `NettyBlockRpcServer` will reading data through BlockManager, > while the block is not cached in memory, the data should be read from DISK > first, then into MEM. I wonder if we implement a mechanism add a message > contains the blockIds that the same as the openBlock message but one loop > ahead, then the `NettyBlockRpcServer ` will load the block ready for transfer > to the reduce side. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16593) Provide a pre-fetch mechanism to accelerate shuffle stage.
[ https://issues.apache.org/jira/browse/SPARK-16593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381391#comment-15381391 ] Apache Spark commented on SPARK-16593: -- User 'f7753' has created a pull request for this issue: https://github.com/apache/spark/pull/14239 > Provide a pre-fetch mechanism to accelerate shuffle stage. > --- > > Key: SPARK-16593 > URL: https://issues.apache.org/jira/browse/SPARK-16593 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Biao Ma >Priority: Minor > Labels: features > > Currently, the `NettyBlockRpcServer` will reading data through BlockManager, > while the block is not cached in memory, the data should be read from DISK > first, then into MEM. I wonder if we implement a mechanism add a message > contains the blockIds that the same as the openBlock message but one loop > ahead, then the `NettyBlockRpcServer ` will load the block ready for transfer > to the reduce side. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
