Hello Spark Users, Someone has suggested by breaking 5-5 unpredictable transformation blocks to Future[ONE STRING ARGUMENT] and claim this can tune the performance. I am wondering this is a use of explicit Future! in Spark?
Sample code is below: def writeData( tableName: String): Future[String] = Future { // some heavy lifting Spark transformations, 5-6 read->transform->load. } writeDataFutures += writeData("dynamicFieldData") writeDataFutures foreach { writeDataFuture => Await.ready(writeDataFuture, Duration.Inf).onComplete { case Success(table) => logger.info(s"All Success") case Failure(e) => e.printStackTrace() } } Please suggest technical doubt. Thanks