[jira] [Updated] (SPARK-26360) Avoid extra validateQuery call in createStreamingWriteSupport
[ https://issues.apache.org/jira/browse/SPARK-26360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wu Wenjie updated SPARK-26360: -- Description: When I'm reading structured streaming source code, I find there is a redundant KafkaWriter.validateQuery() function call in createStreamingWriteSupport func in class `KafkaSourceProvider`. {code:scala} // KafkaSourceProvider.scala override def createStreamingWriteSupport( queryId: String, schema: StructType, mode: OutputMode, options: DataSourceOptions): StreamingWriteSupport = { . // validate once here KafkaWriter.validateQuery(schema.toAttributes, producerParams, topic) // validate twice here new KafkaStreamingWriteSupport(topic, producerParams, schema) } // KafkaStreamingWriteSupport.scala class KafkaStreamingWriteSupport( topic: Option[String], producerParams: ju.Map[String, Object], schema: StructType) extends StreamingWriteSupport { validateQuery(schema.toAttributes, producerParams, topic) } {code} I think we just need to remove one of these two. was: When I'm reading structured streaming source code, I find there is a extra KafkaWriter.validateQuery() function call in createStreamingWriteSupport func in class KafkaSourceProvider. {code:scala} // KafkaSourceProvider.scala override def createStreamingWriteSupport( queryId: String, schema: StructType, mode: OutputMode, options: DataSourceOptions): StreamingWriteSupport = { . // validate once here KafkaWriter.validateQuery(schema.toAttributes, producerParams, topic) // validate twice here new KafkaStreamingWriteSupport(topic, producerParams, schema) } // KafkaStreamingWriteSupport.scala class KafkaStreamingWriteSupport( topic: Option[String], producerParams: ju.Map[String, Object], schema: StructType) extends StreamingWriteSupport { validateQuery(schema.toAttributes, producerParams, topic) } {code} I think we just need to remove one of these two. > Avoid extra validateQuery call in createStreamingWriteSupport > - > > Key: SPARK-26360 > URL: https://issues.apache.org/jira/browse/SPARK-26360 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming >Affects Versions: 2.3.2, 2.4.0 >Reporter: Wu Wenjie >Priority: Trivial > > When I'm reading structured streaming source code, I find there is a > redundant KafkaWriter.validateQuery() function call in > createStreamingWriteSupport func in class `KafkaSourceProvider`. > {code:scala} > // KafkaSourceProvider.scala > override def createStreamingWriteSupport( > queryId: String, > schema: StructType, > mode: OutputMode, > options: DataSourceOptions): StreamingWriteSupport = { >. > // validate once here > KafkaWriter.validateQuery(schema.toAttributes, producerParams, topic) > // validate twice here > new KafkaStreamingWriteSupport(topic, producerParams, schema) > } > // KafkaStreamingWriteSupport.scala > class KafkaStreamingWriteSupport( > topic: Option[String], > producerParams: ju.Map[String, Object], > schema: StructType) > extends StreamingWriteSupport { > validateQuery(schema.toAttributes, producerParams, topic) > > } > {code} > > I think we just need to remove one of these two. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26360) Avoid extra validateQuery call in createStreamingWriteSupport
[ https://issues.apache.org/jira/browse/SPARK-26360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wu Wenjie updated SPARK-26360: -- Shepherd: Wenchen Fan > Avoid extra validateQuery call in createStreamingWriteSupport > - > > Key: SPARK-26360 > URL: https://issues.apache.org/jira/browse/SPARK-26360 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming >Affects Versions: 2.3.2, 2.4.0 >Reporter: Wu Wenjie >Priority: Trivial > > When I'm reading structured streaming source code, I find there is a extra > KafkaWriter.validateQuery() function call in createStreamingWriteSupport func > in class > KafkaSourceProvider. > {code:scala} > // KafkaSourceProvider.scala > override def createStreamingWriteSupport( > queryId: String, > schema: StructType, > mode: OutputMode, > options: DataSourceOptions): StreamingWriteSupport = { >. > // validate once here > KafkaWriter.validateQuery(schema.toAttributes, producerParams, topic) > // validate twice here > new KafkaStreamingWriteSupport(topic, producerParams, schema) > } > // KafkaStreamingWriteSupport.scala > class KafkaStreamingWriteSupport( > topic: Option[String], > producerParams: ju.Map[String, Object], > schema: StructType) > extends StreamingWriteSupport { > validateQuery(schema.toAttributes, producerParams, topic) > > } > {code} > > I think we just need to remove one of these two. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26360) Avoid extra validateQuery call in createStreamingWriteSupport
[ https://issues.apache.org/jira/browse/SPARK-26360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wu Wenjie updated SPARK-26360: -- Shepherd: Sean Owen (was: Wenchen Fan) > Avoid extra validateQuery call in createStreamingWriteSupport > - > > Key: SPARK-26360 > URL: https://issues.apache.org/jira/browse/SPARK-26360 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming >Affects Versions: 2.3.2, 2.4.0 >Reporter: Wu Wenjie >Priority: Trivial > > When I'm reading structured streaming source code, I find there is a extra > KafkaWriter.validateQuery() function call in createStreamingWriteSupport func > in class > KafkaSourceProvider. > {code:scala} > // KafkaSourceProvider.scala > override def createStreamingWriteSupport( > queryId: String, > schema: StructType, > mode: OutputMode, > options: DataSourceOptions): StreamingWriteSupport = { >. > // validate once here > KafkaWriter.validateQuery(schema.toAttributes, producerParams, topic) > // validate twice here > new KafkaStreamingWriteSupport(topic, producerParams, schema) > } > // KafkaStreamingWriteSupport.scala > class KafkaStreamingWriteSupport( > topic: Option[String], > producerParams: ju.Map[String, Object], > schema: StructType) > extends StreamingWriteSupport { > validateQuery(schema.toAttributes, producerParams, topic) > > } > {code} > > I think we just need to remove one of these two. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26360) Avoid extra validateQuery call in createStreamingWriteSupport
[ https://issues.apache.org/jira/browse/SPARK-26360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wu Wenjie updated SPARK-26360: -- Description: When I'm reading structured streaming source code, I find there is a extra KafkaWriter.validateQuery() function call in createStreamingWriteSupport func in class KafkaSourceProvider. {code:scala} // KafkaSourceProvider.scala override def createStreamingWriteSupport( queryId: String, schema: StructType, mode: OutputMode, options: DataSourceOptions): StreamingWriteSupport = { . // validate once here KafkaWriter.validateQuery(schema.toAttributes, producerParams, topic) // validate twice here new KafkaStreamingWriteSupport(topic, producerParams, schema) } // KafkaStreamingWriteSupport.scala class KafkaStreamingWriteSupport( topic: Option[String], producerParams: ju.Map[String, Object], schema: StructType) extends StreamingWriteSupport { validateQuery(schema.toAttributes, producerParams, topic) } {code} I think we just need to remove one of these two. was: When I'm reading structured streaming source code, I find there is a extra KafkaWriter.validateQuery() function call in createStreamingWriteSupport func in class KafkaSourceProvider. {code:scala} // KafkaSourceProvider.scala override def createStreamingWriteSupport( queryId: String, schema: StructType, mode: OutputMode, options: DataSourceOptions): StreamingWriteSupport = { . // validate once here KafkaWriter.validateQuery(schema.toAttributes, producerParams, topic) // validate twice here new KafkaStreamingWriteSupport(topic, producerParams, schema) } // KafkaStreamingWriteSupport.scala class KafkaStreamingWriteSupport( topic: Option[String], producerParams: ju.Map[String, Object], schema: StructType) extends StreamingWriteSupport { validateQuery(schema.toAttributes, producerParams, topic) } {code} I think we just need to remove it. > Avoid extra validateQuery call in createStreamingWriteSupport > - > > Key: SPARK-26360 > URL: https://issues.apache.org/jira/browse/SPARK-26360 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming >Affects Versions: 2.3.2, 2.4.0 >Reporter: Wu Wenjie >Priority: Trivial > > When I'm reading structured streaming source code, I find there is a extra > KafkaWriter.validateQuery() function call in createStreamingWriteSupport func > in class > KafkaSourceProvider. > {code:scala} > // KafkaSourceProvider.scala > override def createStreamingWriteSupport( > queryId: String, > schema: StructType, > mode: OutputMode, > options: DataSourceOptions): StreamingWriteSupport = { >. > // validate once here > KafkaWriter.validateQuery(schema.toAttributes, producerParams, topic) > // validate twice here > new KafkaStreamingWriteSupport(topic, producerParams, schema) > } > // KafkaStreamingWriteSupport.scala > class KafkaStreamingWriteSupport( > topic: Option[String], > producerParams: ju.Map[String, Object], > schema: StructType) > extends StreamingWriteSupport { > validateQuery(schema.toAttributes, producerParams, topic) > > } > {code} > > I think we just need to remove one of these two. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-26360) Avoid extra validateQuery call in createStreamingWriteSupport
Wu Wenjie created SPARK-26360: - Summary: Avoid extra validateQuery call in createStreamingWriteSupport Key: SPARK-26360 URL: https://issues.apache.org/jira/browse/SPARK-26360 Project: Spark Issue Type: Improvement Components: Structured Streaming Affects Versions: 2.4.0, 2.3.2 Reporter: Wu Wenjie When I'm reading structured streaming source code, I find there is a extra KafkaWriter.validateQuery() function call in createStreamingWriteSupport func in class KafkaSourceProvider. {code:scala} // KafkaSourceProvider.scala override def createStreamingWriteSupport( queryId: String, schema: StructType, mode: OutputMode, options: DataSourceOptions): StreamingWriteSupport = { . // validate once here KafkaWriter.validateQuery(schema.toAttributes, producerParams, topic) // validate twice here new KafkaStreamingWriteSupport(topic, producerParams, schema) } // KafkaStreamingWriteSupport.scala class KafkaStreamingWriteSupport( topic: Option[String], producerParams: ju.Map[String, Object], schema: StructType) extends StreamingWriteSupport { validateQuery(schema.toAttributes, producerParams, topic) } {code} I think we just need to remove it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org