Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
dongjoon-hyun commented on PR #46019: URL: https://github.com/apache/spark/pull/46019#issuecomment-2066830650 Merged to branch-3.5. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
dongjoon-hyun commented on PR #46019: URL: https://github.com/apache/spark/pull/46019#issuecomment-2066829697 Thank you all! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
dongjoon-hyun closed pull request #46019: [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated URL: https://github.com/apache/spark/pull/46019 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
dongjoon-hyun commented on PR #46019: URL: https://github.com/apache/spark/pull/46019#issuecomment-2064652257 Hi, @cloud-fan and @HyukjinKwon . How about the AS-IS status? Could you review this once more when you have a chance? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
dongjoon-hyun commented on code in PR #46019:
URL: https://github.com/apache/spark/pull/46019#discussion_r1562817518
##
core/src/main/scala/org/apache/spark/api/python/WriteInputFormatTestDataGenerator.scala:
##
@@ -104,6 +105,7 @@ private[python] class WritableToDoubleArrayConverter
extends Converter[Any, Arra
* This object contains method to generate SequenceFile test data and write it
to a
* given directory (probably a temp directory)
*/
+@deprecated("This class will be move to `test`.", "3.5.2")
Review Comment:
To me, the package name `org/apache/spark/api/python` itself clearly implies
this was added as a part of API at that time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
dongjoon-hyun commented on code in PR #46019:
URL: https://github.com/apache/spark/pull/46019#discussion_r1562817518
##
core/src/main/scala/org/apache/spark/api/python/WriteInputFormatTestDataGenerator.scala:
##
@@ -104,6 +105,7 @@ private[python] class WritableToDoubleArrayConverter
extends Converter[Any, Arra
* This object contains method to generate SequenceFile test data and write it
to a
* given directory (probably a temp directory)
*/
+@deprecated("This class will be move to `test`.", "3.5.2")
Review Comment:
To me, the package name `org/apache/spark/api/python` itself clearly implies
this was added as a part of API at that time and exposed mistakenly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
dongjoon-hyun commented on code in PR #46019:
URL: https://github.com/apache/spark/pull/46019#discussion_r1562814727
##
core/src/main/scala/org/apache/spark/api/python/WriteInputFormatTestDataGenerator.scala:
##
@@ -104,6 +105,7 @@ private[python] class WritableToDoubleArrayConverter
extends Converter[Any, Arra
* This object contains method to generate SequenceFile test data and write it
to a
* given directory (probably a temp directory)
*/
+@deprecated("This class will be move to `test`.", "3.5.2")
Review Comment:
@HyukjinKwon . I understand your point. However, this is **exposed** since
Apache Spark 1.1.0. We don't know if there is someone who take advantage of
this API or not. If we say simply `it's a mistake` and remove **exposed**
methods (Apache Spark 1.1.0 ~ 3.5.x) without deprecation, it would be a
violation of Semantic Versioning. We have `MIMA` check to prevent those kind of
weird situation.
> My concern is that we're marking deprecation, and it sounds like we're
admitting those are actually an API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
dongjoon-hyun commented on code in PR #46019:
URL: https://github.com/apache/spark/pull/46019#discussion_r1562814727
##
core/src/main/scala/org/apache/spark/api/python/WriteInputFormatTestDataGenerator.scala:
##
@@ -104,6 +105,7 @@ private[python] class WritableToDoubleArrayConverter
extends Converter[Any, Arra
* This object contains method to generate SequenceFile test data and write it
to a
* given directory (probably a temp directory)
*/
+@deprecated("This class will be move to `test`.", "3.5.2")
Review Comment:
@HyukjinKwon . I understand your point. However, this is **exposed** since
Apache Spark 1.1.0. We don't know if there is someone take advantage of this
API or not. If we say simply `it's a mistake` and remove **exposed** methods
(Apache Spark 1.1.0 ~ 3.5.x) without deprecation, it would be a violation of
Semantic Versioning. We have `MIMA` check to prevent those kind of weird
situation.
> My concern is that we're marking deprecation, and it sounds like we're
admitting those are actually an API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
panbingkun commented on code in PR #46019:
URL: https://github.com/apache/spark/pull/46019#discussion_r1562211489
##
core/src/main/scala/org/apache/spark/api/python/WriteInputFormatTestDataGenerator.scala:
##
@@ -104,6 +105,7 @@ private[python] class WritableToDoubleArrayConverter
extends Converter[Any, Arra
* This object contains method to generate SequenceFile test data and write it
to a
* given directory (probably a temp directory)
*/
+@deprecated("This class will be move to `test`.", "3.5.2")
Review Comment:
It seems to be only used in `Python related testing`, but it is indeed in
the folder `main` instead of the folder `test`. Can we move it directly without
doing any pre-deprecated operations in version `3.5`?
I'm not quite sure about this rule.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
HyukjinKwon commented on code in PR #46019:
URL: https://github.com/apache/spark/pull/46019#discussion_r1562007347
##
core/src/main/scala/org/apache/spark/api/python/WriteInputFormatTestDataGenerator.scala:
##
@@ -104,6 +105,7 @@ private[python] class WritableToDoubleArrayConverter
extends Converter[Any, Arra
* This object contains method to generate SequenceFile test data and write it
to a
* given directory (probably a temp directory)
*/
+@deprecated("This class will be move to `test`.", "3.5.2")
Review Comment:
H ... things here aren't really an API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
HyukjinKwon commented on code in PR #46019:
URL: https://github.com/apache/spark/pull/46019#discussion_r1562007477
##
core/src/main/scala/org/apache/spark/api/python/WriteInputFormatTestDataGenerator.scala:
##
@@ -104,6 +105,7 @@ private[python] class WritableToDoubleArrayConverter
extends Converter[Any, Arra
* This object contains method to generate SequenceFile test data and write it
to a
* given directory (probably a temp directory)
*/
+@deprecated("This class will be move to `test`.", "3.5.2")
Review Comment:
My concern is that we're marking deprecation, and it sounds like we're
admitting those are actually an API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
dongjoon-hyun commented on PR #46019: URL: https://github.com/apache/spark/pull/46019#issuecomment-2050904083 Thank you for the swift update. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
panbingkun commented on code in PR #46019:
URL: https://github.com/apache/spark/pull/46019#discussion_r1561973227
##
connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisTestUtils.scala:
##
@@ -40,6 +40,7 @@ import org.apache.spark.internal.Logging
*
* PLEASE KEEP THIS FILE UNDER src/main AS PYTHON TESTS NEED ACCESS TO THIS
FILE!
*/
+@deprecated("This class will be move to `test`.", "3.5.2")
private[kinesis] class KinesisTestUtils(streamShardCount: Int = 2) extends
Logging {
Review Comment:
All done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
panbingkun commented on code in PR #46019:
URL: https://github.com/apache/spark/pull/46019#discussion_r1561971976
##
connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisTestUtils.scala:
##
@@ -40,6 +40,7 @@ import org.apache.spark.internal.Logging
*
* PLEASE KEEP THIS FILE UNDER src/main AS PYTHON TESTS NEED ACCESS TO THIS
FILE!
*/
+@deprecated("This class will be move to `test`.", "3.5.2")
private[kinesis] class KinesisTestUtils(streamShardCount: Int = 2) extends
Logging {
Review Comment:
Okay
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
dongjoon-hyun commented on code in PR #46019:
URL: https://github.com/apache/spark/pull/46019#discussion_r1561971654
##
connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisTestUtils.scala:
##
@@ -40,6 +40,7 @@ import org.apache.spark.internal.Logging
*
* PLEASE KEEP THIS FILE UNDER src/main AS PYTHON TESTS NEED ACCESS TO THIS
FILE!
*/
+@deprecated("This class will be move to `test`.", "3.5.2")
private[kinesis] class KinesisTestUtils(streamShardCount: Int = 2) extends
Logging {
Review Comment:
Yes, that needs mark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
panbingkun commented on code in PR #46019:
URL: https://github.com/apache/spark/pull/46019#discussion_r1561971448
##
core/src/main/scala/org/apache/spark/api/python/WriteInputFormatTestDataGenerator.scala:
##
@@ -56,33 +57,39 @@ case class TestWritable(var str: String, var int: Int, var
double: Double) exten
}
}
+@deprecated("This class will be move to `test`.", "3.5.2")
private[python] class TestInputKeyConverter extends Converter[Any, Any] {
override def convert(obj: Any): Char = {
obj.asInstanceOf[IntWritable].get().toChar
}
}
+@deprecated("This class will be move to `test`.", "3.5.2")
private[python] class TestInputValueConverter extends Converter[Any, Any] {
override def convert(obj: Any): ju.List[Double] = {
val m = obj.asInstanceOf[MapWritable]
m.keySet.asScala.map(_.asInstanceOf[DoubleWritable].get()).toSeq.asJava
}
}
+@deprecated("This class will be move to `test`.", "3.5.2")
private[python] class TestOutputKeyConverter extends Converter[Any, Any] {
override def convert(obj: Any): Text = {
new Text(obj.asInstanceOf[Int].toString)
}
}
+@deprecated("This class will be move to `test`.", "3.5.2")
private[python] class TestOutputValueConverter extends Converter[Any, Any] {
Review Comment:
Yeah, according to this principle, we only need to focus on whether the
method is `public`, rather than `internal usage`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
LuciferYang commented on code in PR #46019:
URL: https://github.com/apache/spark/pull/46019#discussion_r1561970178
##
core/src/main/scala/org/apache/spark/api/python/WriteInputFormatTestDataGenerator.scala:
##
@@ -56,33 +57,39 @@ case class TestWritable(var str: String, var int: Int, var
double: Double) exten
}
}
+@deprecated("This class will be move to `test`.", "3.5.2")
private[python] class TestInputKeyConverter extends Converter[Any, Any] {
override def convert(obj: Any): Char = {
obj.asInstanceOf[IntWritable].get().toChar
}
}
+@deprecated("This class will be move to `test`.", "3.5.2")
private[python] class TestInputValueConverter extends Converter[Any, Any] {
override def convert(obj: Any): ju.List[Double] = {
val m = obj.asInstanceOf[MapWritable]
m.keySet.asScala.map(_.asInstanceOf[DoubleWritable].get()).toSeq.asJava
}
}
+@deprecated("This class will be move to `test`.", "3.5.2")
private[python] class TestOutputKeyConverter extends Converter[Any, Any] {
override def convert(obj: Any): Text = {
new Text(obj.asInstanceOf[Int].toString)
}
}
+@deprecated("This class will be move to `test`.", "3.5.2")
private[python] class TestOutputValueConverter extends Converter[Any, Any] {
Review Comment:
`private[python]` probably doesn't need to add `@deprecated`, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
panbingkun commented on code in PR #46019:
URL: https://github.com/apache/spark/pull/46019#discussion_r1561970075
##
connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisTestUtils.scala:
##
@@ -40,6 +40,7 @@ import org.apache.spark.internal.Logging
*
* PLEASE KEEP THIS FILE UNDER src/main AS PYTHON TESTS NEED ACCESS TO THIS
FILE!
*/
+@deprecated("This class will be move to `test`.", "3.5.2")
private[kinesis] class KinesisTestUtils(streamShardCount: Int = 2) extends
Logging {
Review Comment:
Does this class also need to be `marked`?
https://github.com/apache/spark/assets/15246973/7081e2ea-a696-419d-97ea-8ea6d001456c;>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
dongjoon-hyun commented on code in PR #46019:
URL: https://github.com/apache/spark/pull/46019#discussion_r1561969234
##
connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisTestUtils.scala:
##
@@ -40,6 +40,7 @@ import org.apache.spark.internal.Logging
*
* PLEASE KEEP THIS FILE UNDER src/main AS PYTHON TESTS NEED ACCESS TO THIS
FILE!
*/
+@deprecated("This class will be move to `test`.", "3.5.2")
private[kinesis] class KinesisTestUtils(streamShardCount: Int = 2) extends
Logging {
Review Comment:
To @panbingkun , sorry but could you remove the ones on `private[kinesis]`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
LuciferYang commented on code in PR #46019:
URL: https://github.com/apache/spark/pull/46019#discussion_r1561969284
##
connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisTestUtils.scala:
##
@@ -40,6 +40,7 @@ import org.apache.spark.internal.Logging
*
* PLEASE KEEP THIS FILE UNDER src/main AS PYTHON TESTS NEED ACCESS TO THIS
FILE!
*/
+@deprecated("This class will be move to `test`.", "3.5.2")
private[kinesis] class KinesisTestUtils(streamShardCount: Int = 2) extends
Logging {
Review Comment:
> are these APIs? They are `private[kinesis]`
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
dongjoon-hyun commented on code in PR #46019:
URL: https://github.com/apache/spark/pull/46019#discussion_r1561968756
##
connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisTestUtils.scala:
##
@@ -40,6 +40,7 @@ import org.apache.spark.internal.Logging
*
* PLEASE KEEP THIS FILE UNDER src/main AS PYTHON TESTS NEED ACCESS TO THIS
FILE!
*/
+@deprecated("This class will be move to `test`.", "3.5.2")
private[kinesis] class KinesisTestUtils(streamShardCount: Int = 2) extends
Logging {
Review Comment:
Oh, it's my bad. You're right. Only `WriteInputFormatTestDataGenerator` and
`TestWritable ` are the valid ones.
##
connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisTestUtils.scala:
##
@@ -40,6 +40,7 @@ import org.apache.spark.internal.Logging
*
* PLEASE KEEP THIS FILE UNDER src/main AS PYTHON TESTS NEED ACCESS TO THIS
FILE!
*/
+@deprecated("This class will be move to `test`.", "3.5.2")
private[kinesis] class KinesisTestUtils(streamShardCount: Int = 2) extends
Logging {
Review Comment:
To @panbingkun , sorry but could you remove the ones on `private[kinesis]`
and `private[python]`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
dongjoon-hyun commented on code in PR #46019:
URL: https://github.com/apache/spark/pull/46019#discussion_r1561968756
##
connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisTestUtils.scala:
##
@@ -40,6 +40,7 @@ import org.apache.spark.internal.Logging
*
* PLEASE KEEP THIS FILE UNDER src/main AS PYTHON TESTS NEED ACCESS TO THIS
FILE!
*/
+@deprecated("This class will be move to `test`.", "3.5.2")
private[kinesis] class KinesisTestUtils(streamShardCount: Int = 2) extends
Logging {
Review Comment:
Oh, it's my bad. You're right. Only `WriteInputFormatTestDataGenerator` is
the valid one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
cloud-fan commented on code in PR #46019:
URL: https://github.com/apache/spark/pull/46019#discussion_r1561966672
##
connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisTestUtils.scala:
##
@@ -40,6 +40,7 @@ import org.apache.spark.internal.Logging
*
* PLEASE KEEP THIS FILE UNDER src/main AS PYTHON TESTS NEED ACCESS TO THIS
FILE!
*/
+@deprecated("This class will be move to `test`.", "3.5.2")
private[kinesis] class KinesisTestUtils(streamShardCount: Int = 2) extends
Logging {
Review Comment:
are these APIs? They are `private[kinesis]`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
panbingkun commented on PR #46019: URL: https://github.com/apache/spark/pull/46019#issuecomment-2050891561 > BTW, to @panbingkun , we need to wait for this kind of change because we need to collect more opinions than a normal PR. It will take at least 72 hours in general. Okay, don't worry, I accidentally found `this` while working on structured log migration. Thank you very much! ❤️ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
dongjoon-hyun commented on PR #46019: URL: https://github.com/apache/spark/pull/46019#issuecomment-2050880237 BTW, to @panbingkun , we need to wait for this kind of change because we need to collect more opinions than a normal PR. It will take at least 72 hours in general. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
HyukjinKwon commented on PR #46019: URL: https://github.com/apache/spark/pull/46019#issuecomment-2050795814 cc @HeartSaVioR too -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
dongjoon-hyun commented on PR #46019: URL: https://github.com/apache/spark/pull/46019#issuecomment-2050793224 cc @cloud-fan , @HyukjinKwon , @LuciferYang , @zhengruifeng, @yaooqinn , too -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47825][DSTREAMS][3.5] Make `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` deprecated [spark]
panbingkun commented on PR #46019: URL: https://github.com/apache/spark/pull/46019#issuecomment-2050726400 @dongjoon-hyun Is my approach correct? Because it's my `first time` doing this, if there's anything wrong, please point it out. Thank you very much! ❤️ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
