[jira] [Commented] (SPARK-30180) listJars() function display issue.
[
https://issues.apache.org/jira/browse/SPARK-30180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991214#comment-16991214
]
Ankit Raj Boudh commented on SPARK-30180:
-
i am working in this jira, will raise PR soon
> listJars() function display issue.
> --
>
> Key: SPARK-30180
> URL: https://issues.apache.org/jira/browse/SPARK-30180
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.4.4
>Reporter: Ankit Raj Boudh
>Priority: Minor
>
>
> {{scala> sc.listJars()
> res2: Seq[String] = Vector(spark://11.242.181.153:50811/jars/c6%20test.jar)}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-30180) listJars() function display issue.
Ankit Raj Boudh created SPARK-30180:
---
Summary: listJars() function display issue.
Key: SPARK-30180
URL: https://issues.apache.org/jira/browse/SPARK-30180
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.4.4
Reporter: Ankit Raj Boudh
{{scala> sc.listJars()
res2: Seq[String] = Vector(spark://11.242.181.153:50811/jars/c6%20test.jar)}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28367) Kafka connector infinite wait because metadata never updated
[ https://issues.apache.org/jira/browse/SPARK-28367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991202#comment-16991202 ] Jungtaek Lim commented on SPARK-28367: -- Small correction: we're waiting for Kafka 2.4 to leverage KIP-396. > Kafka connector infinite wait because metadata never updated > > > Key: SPARK-28367 > URL: https://issues.apache.org/jira/browse/SPARK-28367 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 2.1.3, 2.2.3, 2.3.3, 2.4.3, 3.0.0 >Reporter: Gabor Somogyi >Priority: Critical > > Spark uses an old and deprecated API named poll(long) which never returns and > stays in live lock if metadata is not updated (for instance when broker > disappears at consumer creation). > I've created a small standalone application to test it and the alternatives: > https://github.com/gaborgsomogyi/kafka-get-assignment -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30169) Eliminate warnings in Kafka connector
[
https://issues.apache.org/jira/browse/SPARK-30169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991201#comment-16991201
]
Jungtaek Lim commented on SPARK-30169:
--
Replacing `poll` is duplicated of
https://issues.apache.org/jira/browse/SPARK-28367 which clearly describes why
it cannot be completed for now (especially comments in PR), and what we are
waiting for (We're waiting for Kafka 2.4 to leverage KIP-396.)
> Eliminate warnings in Kafka connector
> -
>
> Key: SPARK-30169
> URL: https://issues.apache.org/jira/browse/SPARK-30169
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Maxim Gekk
>Priority: Minor
>
> *Eliminate compilation warnings (10) in module external/kafka-0-10*:
> 1.
> spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala
> {code:java}
> Line:108
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> Line:162
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 2.
> spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaDataConsumer.scala
> {code:java}
> Line:206
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 3.
> spark/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala
> {code:java}
> Line:258
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> Line:314
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> Line:476
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 4.
> spark/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
> {code:java}
> Line:60
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:88
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:100
> object ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:178
> method createTopic in object AdminUtils is deprecated (since 1.1.0): This
> method is deprecated and will be replaced by kafka.zk.AdminZkClient.
> {code}
> *Eliminate compilation warnings (13) in module external/kafka-0-10-sql*:
> 1.
> spark/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
> {code}
> Line:74
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 2.
> spark/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala
> {code}
> Line:119
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> Line:382
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> {code}
> 3.
> spark/external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala
> {code}
> Line:84
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:118
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:246
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line: 458
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> Line: 472
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> Line: 581
> object ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:
[jira] [Comment Edited] (SPARK-30169) Eliminate warnings in Kafka connector
[
https://issues.apache.org/jira/browse/SPARK-30169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991197#comment-16991197
]
Aman Omer edited comment on SPARK-30169 at 12/9/19 7:13 AM:
{code:java}
@Deprecated
public ConsumerRecords poll(long timeoutMs) {
return this.poll(this.time.timer(timeoutMs), false);
}
public ConsumerRecords poll(Duration timeout) {
return this.poll(this.time.timer(timeout), true);
}
{code}
{code:java}
default Timer timer(long timeoutMs) {
return new Timer(this, timeoutMs);
}
default Timer timer(Duration timeout) {
return this.timer(timeout.toMillis());
}
{code}
I think it should be ok to convert *poll(0) -> poll(MockTimer.timer(0), false)*
was (Author: aman_omer):
{code:java}
@Deprecated
public ConsumerRecords poll(long timeoutMs) {
return this.poll(this.time.timer(timeoutMs), false);
}
public ConsumerRecords poll(Duration timeout) {
return this.poll(this.time.timer(timeout), true);
}
{code}
{code:java}
default Timer timer(long timeoutMs) {
return new Timer(this, timeoutMs);
}
default Timer timer(Duration timeout) {
return this.timer(timeout.toMillis());
}
{code}
I think it should be ok to convert *poll(0) -> poll(Duration.ZERO)*
> Eliminate warnings in Kafka connector
> -
>
> Key: SPARK-30169
> URL: https://issues.apache.org/jira/browse/SPARK-30169
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Maxim Gekk
>Priority: Minor
>
> *Eliminate compilation warnings (10) in module external/kafka-0-10*:
> 1.
> spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala
> {code:java}
> Line:108
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> Line:162
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 2.
> spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaDataConsumer.scala
> {code:java}
> Line:206
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 3.
> spark/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala
> {code:java}
> Line:258
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> Line:314
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> Line:476
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 4.
> spark/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
> {code:java}
> Line:60
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:88
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:100
> object ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:178
> method createTopic in object AdminUtils is deprecated (since 1.1.0): This
> method is deprecated and will be replaced by kafka.zk.AdminZkClient.
> {code}
> *Eliminate compilation warnings (13) in module external/kafka-0-10-sql*:
> 1.
> spark/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
> {code}
> Line:74
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 2.
> spark/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala
> {code}
> Line:119
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> Line:382
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> {code}
> 3.
> spark/external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala
> {code}
> Line:84
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:118
> class ZkUtils in
[jira] [Commented] (SPARK-30169) Eliminate warnings in Kafka connector
[
https://issues.apache.org/jira/browse/SPARK-30169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991198#comment-16991198
]
Jungtaek Lim commented on SPARK-30169:
--
[~aman_omer]
Look for KIP which introduces poll(Duration).
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-266%3A+Fix+consumer+indefinite+blocking+behavior]
The KIP page clarified poll(0) is not a case where Kafka has been officially
supported. That's actually the hack where we can't do with poll(Duration).
> Eliminate warnings in Kafka connector
> -
>
> Key: SPARK-30169
> URL: https://issues.apache.org/jira/browse/SPARK-30169
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Maxim Gekk
>Priority: Minor
>
> *Eliminate compilation warnings (10) in module external/kafka-0-10*:
> 1.
> spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala
> {code:java}
> Line:108
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> Line:162
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 2.
> spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaDataConsumer.scala
> {code:java}
> Line:206
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 3.
> spark/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala
> {code:java}
> Line:258
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> Line:314
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> Line:476
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 4.
> spark/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
> {code:java}
> Line:60
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:88
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:100
> object ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:178
> method createTopic in object AdminUtils is deprecated (since 1.1.0): This
> method is deprecated and will be replaced by kafka.zk.AdminZkClient.
> {code}
> *Eliminate compilation warnings (13) in module external/kafka-0-10-sql*:
> 1.
> spark/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
> {code}
> Line:74
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 2.
> spark/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala
> {code}
> Line:119
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> Line:382
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> {code}
> 3.
> spark/external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala
> {code}
> Line:84
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:118
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:246
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line: 458
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> Line: 472
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> Line: 581
> object ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please
[jira] [Commented] (SPARK-30169) Eliminate warnings in Kafka connector
[
https://issues.apache.org/jira/browse/SPARK-30169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991197#comment-16991197
]
Aman Omer commented on SPARK-30169:
---
{code:java}
@Deprecated
public ConsumerRecords poll(long timeoutMs) {
return this.poll(this.time.timer(timeoutMs), false);
}
public ConsumerRecords poll(Duration timeout) {
return this.poll(this.time.timer(timeout), true);
}
{code}
{code:java}
default Timer timer(long timeoutMs) {
return new Timer(this, timeoutMs);
}
default Timer timer(Duration timeout) {
return this.timer(timeout.toMillis());
}
{code}
I think it should be ok to convert *poll(0) -> poll(Duration.ZERO)*
> Eliminate warnings in Kafka connector
> -
>
> Key: SPARK-30169
> URL: https://issues.apache.org/jira/browse/SPARK-30169
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Maxim Gekk
>Priority: Minor
>
> *Eliminate compilation warnings (10) in module external/kafka-0-10*:
> 1.
> spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala
> {code:java}
> Line:108
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> Line:162
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 2.
> spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaDataConsumer.scala
> {code:java}
> Line:206
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 3.
> spark/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala
> {code:java}
> Line:258
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> Line:314
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> Line:476
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 4.
> spark/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
> {code:java}
> Line:60
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:88
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:100
> object ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:178
> method createTopic in object AdminUtils is deprecated (since 1.1.0): This
> method is deprecated and will be replaced by kafka.zk.AdminZkClient.
> {code}
> *Eliminate compilation warnings (13) in module external/kafka-0-10-sql*:
> 1.
> spark/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
> {code}
> Line:74
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 2.
> spark/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala
> {code}
> Line:119
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> Line:382
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> {code}
> 3.
> spark/external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala
> {code}
> Line:84
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:118
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:246
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line: 458
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> Line: 472
> method poll in class KafkaConsumer is deprecated: see
[jira] [Updated] (SPARK-30169) Eliminate warnings in Kafka connector
[
https://issues.apache.org/jira/browse/SPARK-30169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aman Omer updated SPARK-30169:
--
Description:
*Eliminate compilation warnings (10) in module external/kafka-0-10*:
1.
spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala
{code:java}
Line:108
method poll in class KafkaConsumer is deprecated: see corresponding Javadoc for
more information.
Line:162
method poll in class KafkaConsumer is deprecated: see corresponding Javadoc for
more information.
{code}
2.
spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaDataConsumer.scala
{code:java}
Line:206
method poll in class KafkaConsumer is deprecated: see corresponding Javadoc for
more information.
{code}
3.
spark/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala
{code:java}
Line:258
method poll in trait Consumer is deprecated: see corresponding Javadoc for more
information.
Line:314
method poll in trait Consumer is deprecated: see corresponding Javadoc for more
information.
Line:476
method poll in class KafkaConsumer is deprecated: see corresponding Javadoc for
more information.
{code}
4.
spark/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
{code:java}
Line:60
class ZkUtils in package utils is deprecated (since 2.0.0): This is an internal
class that is no longer used by Kafka and will be removed in a future release.
Please use org.apache.kafka.clients.admin.AdminClient instead.
Line:88
class ZkUtils in package utils is deprecated (since 2.0.0): This is an internal
class that is no longer used by Kafka and will be removed in a future release.
Please use org.apache.kafka.clients.admin.AdminClient instead.
Line:100
object ZkUtils in package utils is deprecated (since 2.0.0): This is an
internal class that is no longer used by Kafka and will be removed in a future
release. Please use org.apache.kafka.clients.admin.AdminClient instead.
Line:178
method createTopic in object AdminUtils is deprecated (since 1.1.0): This
method is deprecated and will be replaced by kafka.zk.AdminZkClient.
{code}
*Eliminate compilation warnings (13) in module external/kafka-0-10-sql*:
1.
spark/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
{code}
Line:74
method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
for more information.
{code}
2.
spark/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala
{code}
Line:119
method poll in trait Consumer is deprecated: see corresponding Javadoc for more
information.
Line:382
method poll in trait Consumer is deprecated: see corresponding Javadoc for more
information.
{code}
3.
spark/external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala
{code}
Line:84
class ZkUtils in package utils is deprecated (since 2.0.0): This is an internal
class that is no longer used by Kafka and will be removed in a future release.
Please use org.apache.kafka.clients.admin.AdminClient instead.
Line:118
class ZkUtils in package utils is deprecated (since 2.0.0): This is an internal
class that is no longer used by Kafka and will be removed in a future release.
Please use org.apache.kafka.clients.admin.AdminClient instead.
Line:246
class ZkUtils in package utils is deprecated (since 2.0.0): This is an internal
class that is no longer used by Kafka and will be removed in a future release.
Please use org.apache.kafka.clients.admin.AdminClient instead.
Line: 458
method poll in class KafkaConsumer is deprecated: see corresponding Javadoc for
more information.
Line: 472
method poll in class KafkaConsumer is deprecated: see corresponding Javadoc for
more information.
Line: 581
object ZkUtils in package utils is deprecated (since 2.0.0): This is an
internal class that is no longer used by Kafka and will be removed in a future
release. Please use org.apache.kafka.clients.admin.AdminClient instead.
Line: 582
object ZkUtils in package utils is deprecated (since 2.0.0): This is an
internal class that is no longer used by Kafka and will be removed in a future
release. Please use org.apache.kafka.clients.admin.AdminClient instead.
Line: 583
object ZkUtils in package utils is deprecated (since 2.0.0): This is an
internal class that is no longer used by Kafka and will be removed in a future
release. Please use org.apache.kafka.clients.admin.AdminClient instead.
Line: 583
object ZkUtils in package utils is deprecated (since 2.0.0): This is an
internal class that is no longer used by Kafka and will be removed in a future
release. Please use org.apache.kafka.clients.admin.AdminClient instead.
Line: 607
object ZkUtils in package utils is deprecated (since 2.0.0): This is an
internal class that is no longer used by Kafka and will be
[jira] [Commented] (SPARK-30169) Eliminate warnings in Kafka connector
[
https://issues.apache.org/jira/browse/SPARK-30169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991196#comment-16991196
]
Jungtaek Lim commented on SPARK-30169:
--
Note that this is not just replacing old deprecated method with new one. Kafka
made a huge change between old poll(long) and new poll(Duration).
Former doesn't count the duration for metadata update into account - it's
blocking though we provide timeout (that's how poll(0) works and the intention
of why we call poll(0)). Latter takes metadata update into account of timeout,
so we should give higher enough timeout to new poll. Simply migrating long to
Duration would result in unexpected behavior.
That's why we haven't dealt with "deprecated" in spark-sql-kafka module. We've
already filed couple of JIRA issues to deal with `poll` and `ZkUtils`. (Btw, it
would be great for anyone to deal with ZkUtils as we got lost and don't work
for such issue for now.)
> Eliminate warnings in Kafka connector
> -
>
> Key: SPARK-30169
> URL: https://issues.apache.org/jira/browse/SPARK-30169
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Maxim Gekk
>Priority: Minor
>
> *Eliminate compilation warnings (10) in module external/kafka-0-10*:
> 1.
> /home/root1/spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala
> {code:java}
> Line:108
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> Line:162
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 2.
> /home/root1/spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaDataConsumer.scala
> {code:java}
> Line:206
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 3.
> /home/root1/spark/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala
> {code:java}
> Line:258
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> Line:314
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> Line:476
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 4.
> /home/root1/spark/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
> {code:java}
> Line:60
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:88
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:100
> object ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:178
> method createTopic in object AdminUtils is deprecated (since 1.1.0): This
> method is deprecated and will be replaced by kafka.zk.AdminZkClient.
> {code}
> *Eliminate compilation warnings (13) in module external/kafka-0-10-sql*:
> 1.
> /home/root1/spark/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
> {code}
> Line:74
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 2.
> /home/root1/spark/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala
> {code}
> Line:119
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> Line:382
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> {code}
> 3.
> /home/root1/spark/external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala
> {code}
> Line:84
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:118
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future release. Please use org.apache.kafka.clients.admin.AdminClient instead.
> Line:246
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer
[jira] [Updated] (SPARK-30169) Eliminate warnings in Kafka connector
[
https://issues.apache.org/jira/browse/SPARK-30169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aman Omer updated SPARK-30169:
--
Description:
*Eliminate compilation warnings (10) in module external/kafka-0-10*:
1.
/home/root1/spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala
{code:java}
Line:108
method poll in class KafkaConsumer is deprecated: see corresponding Javadoc for
more information.
Line:162
method poll in class KafkaConsumer is deprecated: see corresponding Javadoc for
more information.
{code}
2.
/home/root1/spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaDataConsumer.scala
{code:java}
Line:206
method poll in class KafkaConsumer is deprecated: see corresponding Javadoc for
more information.
{code}
3.
/home/root1/spark/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala
{code:java}
Line:258
method poll in trait Consumer is deprecated: see corresponding Javadoc for more
information.
Line:314
method poll in trait Consumer is deprecated: see corresponding Javadoc for more
information.
Line:476
method poll in class KafkaConsumer is deprecated: see corresponding Javadoc for
more information.
{code}
4.
/home/root1/spark/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
{code:java}
Line:60
class ZkUtils in package utils is deprecated (since 2.0.0): This is an internal
class that is no longer used by Kafka and will be removed in a future release.
Please use org.apache.kafka.clients.admin.AdminClient instead.
Line:88
class ZkUtils in package utils is deprecated (since 2.0.0): This is an internal
class that is no longer used by Kafka and will be removed in a future release.
Please use org.apache.kafka.clients.admin.AdminClient instead.
Line:100
object ZkUtils in package utils is deprecated (since 2.0.0): This is an
internal class that is no longer used by Kafka and will be removed in a future
release. Please use org.apache.kafka.clients.admin.AdminClient instead.
Line:178
method createTopic in object AdminUtils is deprecated (since 1.1.0): This
method is deprecated and will be replaced by kafka.zk.AdminZkClient.
{code}
*Eliminate compilation warnings (13) in module external/kafka-0-10-sql*:
1.
/home/root1/spark/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
{code}
Line:74
method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
for more information.
{code}
2.
/home/root1/spark/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala
{code}
Line:119
method poll in trait Consumer is deprecated: see corresponding Javadoc for more
information.
Line:382
method poll in trait Consumer is deprecated: see corresponding Javadoc for more
information.
{code}
3.
/home/root1/spark/external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala
{code}
Line:84
class ZkUtils in package utils is deprecated (since 2.0.0): This is an internal
class that is no longer used by Kafka and will be removed in a future release.
Please use org.apache.kafka.clients.admin.AdminClient instead.
Line:118
class ZkUtils in package utils is deprecated (since 2.0.0): This is an internal
class that is no longer used by Kafka and will be removed in a future release.
Please use org.apache.kafka.clients.admin.AdminClient instead.
Line:246
class ZkUtils in package utils is deprecated (since 2.0.0): This is an internal
class that is no longer used by Kafka and will be removed in a future release.
Please use org.apache.kafka.clients.admin.AdminClient instead.
Line: 458
method poll in class KafkaConsumer is deprecated: see corresponding Javadoc for
more information.
Line: 472
method poll in class KafkaConsumer is deprecated: see corresponding Javadoc for
more information.
Line: 581
object ZkUtils in package utils is deprecated (since 2.0.0): This is an
internal class that is no longer used by Kafka and will be removed in a future
release. Please use org.apache.kafka.clients.admin.AdminClient instead.
Line: 582
object ZkUtils in package utils is deprecated (since 2.0.0): This is an
internal class that is no longer used by Kafka and will be removed in a future
release. Please use org.apache.kafka.clients.admin.AdminClient instead.
Line: 583
object ZkUtils in package utils is deprecated (since 2.0.0): This is an
internal class that is no longer used by Kafka and will be removed in a future
release. Please use org.apache.kafka.clients.admin.AdminClient instead.
Line: 583
object ZkUtils in package utils is deprecated (since 2.0.0): This is an
internal class that is no longer used by Kafka and will be removed in a future
release. Please use org.apache.kafka.clients.admin.AdminClient instead.
Line: 607
object ZkUtils in package utils is deprecated
[jira] [Commented] (SPARK-30179) Improve test in SingleSessionSuite
[ https://issues.apache.org/jira/browse/SPARK-30179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991186#comment-16991186 ] Yuming Wang commented on SPARK-30179: - Example: https://github.com/apache/spark/blob/58be82ad4b98fc17e821e916e69e77a6aa36209d/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala#L605-L614 > Improve test in SingleSessionSuite > -- > > Key: SPARK-30179 > URL: https://issues.apache.org/jira/browse/SPARK-30179 > Project: Spark > Issue Type: Bug > Components: Tests >Affects Versions: 3.0.0 >Reporter: Yuming Wang >Priority: Major > > https://github.com/apache/spark/blob/58be82ad4b98fc17e821e916e69e77a6aa36209d/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala#L782-L824 > We should also verify the UDF works. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-30179) Improve test in SingleSessionSuite
Yuming Wang created SPARK-30179: --- Summary: Improve test in SingleSessionSuite Key: SPARK-30179 URL: https://issues.apache.org/jira/browse/SPARK-30179 Project: Spark Issue Type: Bug Components: Tests Affects Versions: 3.0.0 Reporter: Yuming Wang https://github.com/apache/spark/blob/58be82ad4b98fc17e821e916e69e77a6aa36209d/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala#L782-L824 We should also verify the UDF works. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30169) Eliminate warnings in Kafka connector
[
https://issues.apache.org/jira/browse/SPARK-30169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aman Omer updated SPARK-30169:
--
Description:
Eliminate compilation warnings (10) in module external/kafka-0-10:
1.
/home/root1/spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala
{code:java}
Line:108
method poll in class KafkaConsumer is deprecated: see corresponding Javadoc for
more information.
Line:162
method poll in class KafkaConsumer is deprecated: see corresponding Javadoc for
more information.
{code}
2.
/home/root1/spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaDataConsumer.scala
{code:java}
Line:206
method poll in class KafkaConsumer is deprecated: see corresponding Javadoc for
more information.
{code}
3.
/home/root1/spark/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala
{code:java}
Line:258
method poll in trait Consumer is deprecated: see corresponding Javadoc for more
information.
Line:314
method poll in trait Consumer is deprecated: see corresponding Javadoc for more
information.
Line:476
method poll in class KafkaConsumer is deprecated: see corresponding Javadoc for
more information.
{code}
4.
/home/root1/spark/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
{code:java}
Line:60
class ZkUtils in package utils is deprecated (since 2.0.0): This is an internal
class that is no longer used by Kafka and will be removed in a future release.
Please use org.apache.kafka.clients.admin.AdminClient instead.
Line:88
class ZkUtils in package utils is deprecated (since 2.0.0): This is an internal
class that is no longer used by Kafka and will be removed in a future release.
Please use org.apache.kafka.clients.admin.AdminClient instead.
Line:100
object ZkUtils in package utils is deprecated (since 2.0.0): This is an
internal class that is no longer used by Kafka and will be removed in a future
release. Please use org.apache.kafka.clients.admin.AdminClient instead.
Line:178
method createTopic in object AdminUtils is deprecated (since 1.1.0): This
method is deprecated and will be replaced by kafka.zk.AdminZkClient.
{code}
was:
Eliminate compilation warnings in the files:
{code}
external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala
external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/DirectKafkaInputDStream.scala
external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaDataConsumer.scala
external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala
external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala
{code}
> Eliminate warnings in Kafka connector
> -
>
> Key: SPARK-30169
> URL: https://issues.apache.org/jira/browse/SPARK-30169
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Maxim Gekk
>Priority: Minor
>
> Eliminate compilation warnings (10) in module external/kafka-0-10:
> 1.
> /home/root1/spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala
> {code:java}
> Line:108
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> Line:162
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 2.
> /home/root1/spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaDataConsumer.scala
> {code:java}
> Line:206
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 3.
> /home/root1/spark/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala
> {code:java}
> Line:258
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> Line:314
> method poll in trait Consumer is deprecated: see corresponding Javadoc for
> more information.
> Line:476
> method poll in class KafkaConsumer is deprecated: see corresponding Javadoc
> for more information.
> {code}
> 4.
> /home/root1/spark/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
> {code:java}
> Line:60
> class ZkUtils in package utils is deprecated (since 2.0.0): This is an
> internal class that is no longer used by Kafka and will be removed in a
> future
[jira] [Resolved] (SPARK-29883) Implement a helper method for aliasing bool_and() and bool_or()
[
https://issues.apache.org/jira/browse/SPARK-29883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan resolved SPARK-29883.
-
Fix Version/s: 3.0.0
Resolution: Fixed
Issue resolved by pull request 26712
[https://github.com/apache/spark/pull/26712]
> Implement a helper method for aliasing bool_and() and bool_or()
> ---
>
> Key: SPARK-29883
> URL: https://issues.apache.org/jira/browse/SPARK-29883
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Xiao Li
>Assignee: Aman Omer
>Priority: Major
> Fix For: 3.0.0
>
>
>
> A general issue in error message when the function name is just an alias name
> of the actual built-in function. For example, every is an alias of bool_and
> in Spark 3.0
> {code:java}
> cannot resolve 'every('true')' due to data type mismatch: Input to function
> 'every' should have been boolean, but it's [string].; line 1 pos 7
> {code}
> {code:java}
> cannot resolve 'bool_and('true')' due to data type mismatch: Input to
> function 'bool_and' should have been boolean, but it's [string].; line 1 pos
> 7{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-29883) Implement a helper method for aliasing bool_and() and bool_or()
[
https://issues.apache.org/jira/browse/SPARK-29883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan reassigned SPARK-29883:
---
Assignee: Aman Omer
> Implement a helper method for aliasing bool_and() and bool_or()
> ---
>
> Key: SPARK-29883
> URL: https://issues.apache.org/jira/browse/SPARK-29883
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Xiao Li
>Assignee: Aman Omer
>Priority: Major
>
>
> A general issue in error message when the function name is just an alias name
> of the actual built-in function. For example, every is an alias of bool_and
> in Spark 3.0
> {code:java}
> cannot resolve 'every('true')' due to data type mismatch: Input to function
> 'every' should have been boolean, but it's [string].; line 1 pos 7
> {code}
> {code:java}
> cannot resolve 'bool_and('true')' due to data type mismatch: Input to
> function 'bool_and' should have been boolean, but it's [string].; line 1 pos
> 7{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26346) Upgrade parquet to 1.11.0
[ https://issues.apache.org/jira/browse/SPARK-26346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991165#comment-16991165 ] t oo commented on SPARK-26346: -- https://www.apache.org/dist/parquet/apache-parquet-1.11.0/ http://mail-archives.apache.org/mod_mbox/parquet-dev/201912.mbox/browser. released > Upgrade parquet to 1.11.0 > - > > Key: SPARK-26346 > URL: https://issues.apache.org/jira/browse/SPARK-26346 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.0.0 >Reporter: Yuming Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Issue Comment Deleted] (SPARK-30171) Eliminate warnings: part2
[
https://issues.apache.org/jira/browse/SPARK-30171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ankit Raj Boudh updated SPARK-30171:
Comment: was deleted
(was: i will raise PR for this)
> Eliminate warnings: part2
> -
>
> Key: SPARK-30171
> URL: https://issues.apache.org/jira/browse/SPARK-30171
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: ABHISHEK KUMAR GUPTA
>Priority: Minor
>
> AvroFunctionsSuite.scala
> Warning:Warning:line (41)method to_avro in package avro is deprecated (since
> 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro' instead.
> val avroDF = df.select(to_avro('id).as("a"), to_avro('str).as("b"))
> Warning:Warning:line (41)method to_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro'
> instead.
> val avroDF = df.select(to_avro('id).as("a"), to_avro('str).as("b"))
> Warning:Warning:line (54)method from_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
> instead.
> checkAnswer(avroDF.select(from_avro('a, avroTypeLong), from_avro('b,
> avroTypeStr)), df)
> Warning:Warning:line (54)method from_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
> instead.
> checkAnswer(avroDF.select(from_avro('a, avroTypeLong), from_avro('b,
> avroTypeStr)), df)
> Warning:Warning:line (59)method to_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro'
> instead.
> val avroStructDF = df.select(to_avro('struct).as("avro"))
> Warning:Warning:line (70)method from_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
> instead.
> checkAnswer(avroStructDF.select(from_avro('avro, avroTypeStruct)), df)
> Warning:Warning:line (76)method to_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro'
> instead.
> val avroStructDF = df.select(to_avro('struct).as("avro"))
> Warning:Warning:line (118)method to_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro'
> instead.
> val readBackOne = dfOne.select(to_avro($"array").as("avro"))
> Warning:Warning:line (119)method from_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
> instead.
> .select(from_avro($"avro", avroTypeArrStruct).as("array"))
> AvroPartitionReaderFactory.scala
> Warning:Warning:line (64)value ignoreExtension in class AvroOptions is
> deprecated (since 3.0): Use the general data source option pathGlobFilter for
> filtering file names
> if (parsedOptions.ignoreExtension ||
> partitionedFile.filePath.endsWith(".avro")) {
> AvroFileFormat.scala
> Warning:Warning:line (98)value ignoreExtension in class AvroOptions is
> deprecated (since 3.0): Use the general data source option pathGlobFilter for
> filtering file names
> if (parsedOptions.ignoreExtension || file.filePath.endsWith(".avro")) {
> AvroUtils.scala
> Warning:Warning:line (55)value ignoreExtension in class AvroOptions is
> deprecated (since 3.0): Use the general data source option pathGlobFilter for
> filtering file names
> inferAvroSchemaFromFiles(files, conf, parsedOptions.ignoreExtension,
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30171) Eliminate warnings: part2
[
https://issues.apache.org/jira/browse/SPARK-30171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991164#comment-16991164
]
pavithra ramachandran commented on SPARK-30171:
---
i will work on this
> Eliminate warnings: part2
> -
>
> Key: SPARK-30171
> URL: https://issues.apache.org/jira/browse/SPARK-30171
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: ABHISHEK KUMAR GUPTA
>Priority: Minor
>
> AvroFunctionsSuite.scala
> Warning:Warning:line (41)method to_avro in package avro is deprecated (since
> 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro' instead.
> val avroDF = df.select(to_avro('id).as("a"), to_avro('str).as("b"))
> Warning:Warning:line (41)method to_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro'
> instead.
> val avroDF = df.select(to_avro('id).as("a"), to_avro('str).as("b"))
> Warning:Warning:line (54)method from_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
> instead.
> checkAnswer(avroDF.select(from_avro('a, avroTypeLong), from_avro('b,
> avroTypeStr)), df)
> Warning:Warning:line (54)method from_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
> instead.
> checkAnswer(avroDF.select(from_avro('a, avroTypeLong), from_avro('b,
> avroTypeStr)), df)
> Warning:Warning:line (59)method to_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro'
> instead.
> val avroStructDF = df.select(to_avro('struct).as("avro"))
> Warning:Warning:line (70)method from_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
> instead.
> checkAnswer(avroStructDF.select(from_avro('avro, avroTypeStruct)), df)
> Warning:Warning:line (76)method to_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro'
> instead.
> val avroStructDF = df.select(to_avro('struct).as("avro"))
> Warning:Warning:line (118)method to_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro'
> instead.
> val readBackOne = dfOne.select(to_avro($"array").as("avro"))
> Warning:Warning:line (119)method from_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
> instead.
> .select(from_avro($"avro", avroTypeArrStruct).as("array"))
> AvroPartitionReaderFactory.scala
> Warning:Warning:line (64)value ignoreExtension in class AvroOptions is
> deprecated (since 3.0): Use the general data source option pathGlobFilter for
> filtering file names
> if (parsedOptions.ignoreExtension ||
> partitionedFile.filePath.endsWith(".avro")) {
> AvroFileFormat.scala
> Warning:Warning:line (98)value ignoreExtension in class AvroOptions is
> deprecated (since 3.0): Use the general data source option pathGlobFilter for
> filtering file names
> if (parsedOptions.ignoreExtension || file.filePath.endsWith(".avro")) {
> AvroUtils.scala
> Warning:Warning:line (55)value ignoreExtension in class AvroOptions is
> deprecated (since 3.0): Use the general data source option pathGlobFilter for
> filtering file names
> inferAvroSchemaFromFiles(files, conf, parsedOptions.ignoreExtension,
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30176) Eliminate warnings: part 6
[
https://issues.apache.org/jira/browse/SPARK-30176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jobit mathew updated SPARK-30176:
-
Description:
sql/core/src/test/scala/org/apache/spark/sql/DatasetAggregatorSuite.scala
{code:java}
Warning:Warning:line (32)java: org.apache.spark.sql.expressions.javalang.typed
in org.apache.spark.sql.expressions.javalang has been deprecated
Warning:Warning:line (91)java:
org.apache.spark.sql.expressions.javalang.typed in
org.apache.spark.sql.expressions.javalang has been deprecated
Warning:Warning:line (100)java:
org.apache.spark.sql.expressions.javalang.typed in
org.apache.spark.sql.expressions.javalang has been deprecated
Warning:Warning:line (109)java:
org.apache.spark.sql.expressions.javalang.typed in
org.apache.spark.sql.expressions.javalang has been deprecated
Warning:Warning:line (118)java:
org.apache.spark.sql.expressions.javalang.typed in
org.apache.spark.sql.expressions.javalang has been deprecated
{code}
sql/core/src/test/scala/org/apache/spark/sql/DatasetBenchmark.scala
{code:java}
Warning:Warning:line (242)object typed in package scalalang is deprecated
(since 3.0.0): please use untyped builtin aggregate functions.
df.as[Data].select(typed.sumLong((d: Data) =>
d.l)).queryExecution.toRdd.foreach(_ => ())
{code}
sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala
{code:java}
Warning:Warning:line (714)method from_utc_timestamp in object functions is
deprecated (since 3.0.0): This function is deprecated and will be removed in
future versions.
df.select(from_utc_timestamp(col("a"), "PST")),
Warning:Warning:line (719)method from_utc_timestamp in object functions is
deprecated (since 3.0.0): This function is deprecated and will be removed in
future versions.
df.select(from_utc_timestamp(col("b"), "PST")),
Warning:Warning:line (725)method from_utc_timestamp in object functions is
deprecated (since 3.0.0): This function is deprecated and will be removed in
future versions.
df.select(from_utc_timestamp(col("a"), "PST")).collect()
Warning:Warning:line (737)method from_utc_timestamp in object functions is
deprecated (since 3.0.0): This function is deprecated and will be removed in
future versions.
df.select(from_utc_timestamp(col("a"), col("c"))),
Warning:Warning:line (742)method from_utc_timestamp in object functions is
deprecated (since 3.0.0): This function is deprecated and will be removed in
future versions.
df.select(from_utc_timestamp(col("b"), col("c"))),
Warning:Warning:line (756)method to_utc_timestamp in object functions is
deprecated (since 3.0.0): This function is deprecated and will be removed in
future versions.
df.select(to_utc_timestamp(col("a"), "PST")),
Warning:Warning:line (761)method to_utc_timestamp in object functions is
deprecated (since 3.0.0): This function is deprecated and will be removed in
future versions.
df.select(to_utc_timestamp(col("b"), "PST")),
Warning:Warning:line (767)method to_utc_timestamp in object functions is
deprecated (since 3.0.0): This function is deprecated and will be removed in
future versions.
df.select(to_utc_timestamp(col("a"), "PST")).collect()
Warning:Warning:line (779)method to_utc_timestamp in object functions is
deprecated (since 3.0.0): This function is deprecated and will be removed in
future versions.
df.select(to_utc_timestamp(col("a"), col("c"))),
Warning:Warning:line (784)method to_utc_timestamp in object functions is
deprecated (since 3.0.0): This function is deprecated and will be removed in
future versions.
df.select(to_utc_timestamp(col("b"), col("c"))),
{code}
sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala
{code:java}
Warning:Warning:line (241)method merge in object Row is deprecated (since
3.0.0): This method is deprecated and will be removed in future versions.
testData.rdd.flatMap(row => Seq.fill(16)(Row.merge(row,
row))).collect().toSeq)
{code}
sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
{code:java}
Warning:Warning:line (787)method merge in object Row is deprecated (since
3.0.0): This method is deprecated and will be removed in future versions.
row => Seq.fill(16)(Row.merge(row, row))).collect().toSeq)
{code}
sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala
{code:java}
Warning:Warning:line (332)constructor ExpressionInfo in class ExpressionInfo
is deprecated: see corresponding Javadoc for more information.
new ExpressionInfo("noClass", "myDb", "myFunction", "usage", "extended
usage"),
Warning:Warning:line (729)constructor ExpressionInfo in class
ExpressionInfo is deprecated: see corresponding Javadoc for more information.
new ExpressionInfo("noClass", "myDb", "myFunction2", "usage", "extended
usage"),
[jira] [Commented] (SPARK-30176) Eliminate warnings: part 6
[
https://issues.apache.org/jira/browse/SPARK-30176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991151#comment-16991151
]
Rakesh Raushan commented on SPARK-30176:
i will work on this.
> Eliminate warnings: part 6
> --
>
> Key: SPARK-30176
> URL: https://issues.apache.org/jira/browse/SPARK-30176
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: jobit mathew
>Priority: Minor
>
>
> sql/core/src/test/scala/org/apache/spark/sql/DatasetAggregatorSuite.scala
> {code:java}
> {code}
> sql/core/src/test/scala/org/apache/spark/sql/DatasetBenchmark.scala
> {code:java}
> {code}
> sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala
> {code:java}
> {code}
> sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala
> {code:java}
> {code}
> sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
> {code:java}
> {code}
>
> sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala
> {code:java}
> {code}
>
> sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
> {code:java}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29883) Implement a helper method for aliasing bool_and() and bool_or()
[
https://issues.apache.org/jira/browse/SPARK-29883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aman Omer updated SPARK-29883:
--
Summary: Implement a helper method for aliasing bool_and() and bool_or()
(was: Improve error messages when function name is an alias)
> Implement a helper method for aliasing bool_and() and bool_or()
> ---
>
> Key: SPARK-29883
> URL: https://issues.apache.org/jira/browse/SPARK-29883
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Xiao Li
>Priority: Major
>
>
> A general issue in error message when the function name is just an alias name
> of the actual built-in function. For example, every is an alias of bool_and
> in Spark 3.0
> {code:java}
> cannot resolve 'every('true')' due to data type mismatch: Input to function
> 'every' should have been boolean, but it's [string].; line 1 pos 7
> {code}
> {code:java}
> cannot resolve 'bool_and('true')' due to data type mismatch: Input to
> function 'bool_and' should have been boolean, but it's [string].; line 1 pos
> 7{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30176) Eliminate warnings: part 6
[
https://issues.apache.org/jira/browse/SPARK-30176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jobit mathew updated SPARK-30176:
-
Description:
sql/core/src/test/scala/org/apache/spark/sql/DatasetAggregatorSuite.scala
{code:java}
{code}
sql/core/src/test/scala/org/apache/spark/sql/DatasetBenchmark.scala
{code:java}
{code}
sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala
{code:java}
{code}
sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala
{code:java}
{code}
sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
{code:java}
{code}
sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala
{code:java}
{code}
sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
{code:java}
{code}
was:
sql/core/src/test/scala/org/apache/spark/sql/DatasetAggregatorSuite.scala
sql/core/src/test/scala/org/apache/spark/sql/DatasetBenchmark.scala
sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala
sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala
sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala
sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
> Eliminate warnings: part 6
> --
>
> Key: SPARK-30176
> URL: https://issues.apache.org/jira/browse/SPARK-30176
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: jobit mathew
>Priority: Minor
>
>
> sql/core/src/test/scala/org/apache/spark/sql/DatasetAggregatorSuite.scala
> {code:java}
> {code}
> sql/core/src/test/scala/org/apache/spark/sql/DatasetBenchmark.scala
> {code:java}
> {code}
> sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala
> {code:java}
> {code}
> sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala
> {code:java}
> {code}
> sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
> {code:java}
> {code}
>
> sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala
> {code:java}
> {code}
>
> sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
> {code:java}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30175) Eliminate warnings: part 5
[
https://issues.apache.org/jira/browse/SPARK-30175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jobit mathew updated SPARK-30175:
-
Description:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/WriteToMicroBatchDataSource.scala
{code:java}
Warning:Warning:line (36)class WriteToDataSourceV2 in package v2 is deprecated
(since 2.4.0): Use specific logical plans like AppendData instead
def createPlan(batchId: Long): WriteToDataSourceV2 = {
Warning:Warning:line (37)class WriteToDataSourceV2 in package v2 is
deprecated (since 2.4.0): Use specific logical plans like AppendData instead
WriteToDataSourceV2(new MicroBatchWrite(batchId, write), query)
{code}
sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala
{code:java}
Warning:Warning:line (703)a pure expression does nothing in statement
position; multiline expressions might require enclosing parentheses
q1
{code}
sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationSuite.scala
{code:java}
Warning:Warning:line (285)object typed in package scalalang is deprecated
(since 3.0.0): please use untyped builtin aggregate functions.
val aggregated = inputData.toDS().groupByKey(_._1).agg(typed.sumLong(_._2))
{code}
was:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/WriteToMicroBatchDataSource.scala
sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala
sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationSuite.scala
> Eliminate warnings: part 5
> --
>
> Key: SPARK-30175
> URL: https://issues.apache.org/jira/browse/SPARK-30175
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: jobit mathew
>Priority: Minor
>
> sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/WriteToMicroBatchDataSource.scala
> {code:java}
> Warning:Warning:line (36)class WriteToDataSourceV2 in package v2 is
> deprecated (since 2.4.0): Use specific logical plans like AppendData instead
> def createPlan(batchId: Long): WriteToDataSourceV2 = {
> Warning:Warning:line (37)class WriteToDataSourceV2 in package v2 is
> deprecated (since 2.4.0): Use specific logical plans like AppendData instead
> WriteToDataSourceV2(new MicroBatchWrite(batchId, write), query)
> {code}
> sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala
> {code:java}
> Warning:Warning:line (703)a pure expression does nothing in statement
> position; multiline expressions might require enclosing parentheses
> q1
> {code}
> sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationSuite.scala
> {code:java}
> Warning:Warning:line (285)object typed in package scalalang is deprecated
> (since 3.0.0): please use untyped builtin aggregate functions.
> val aggregated =
> inputData.toDS().groupByKey(_._1).agg(typed.sumLong(_._2))
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30177) Eliminate warnings: part7
[ https://issues.apache.org/jira/browse/SPARK-30177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991150#comment-16991150 ] Ankit Raj Boudh commented on SPARK-30177: - I will raise PR for this > Eliminate warnings: part7 > - > > Key: SPARK-30177 > URL: https://issues.apache.org/jira/browse/SPARK-30177 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.0.0 >Reporter: ABHISHEK KUMAR GUPTA >Priority: Minor > > /mllib/src/test/scala/org/apache/spark/ml/clustering/BisectingKMeansSuite.scala > Warning:Warning:line (108)method computeCost in class > BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated > and will be removed in future versions. Use ClusteringEvaluator instead. You > can also get the cost on the training dataset in the summary. > assert(model.computeCost(dataset) < 0.1) > Warning:Warning:line (135)method computeCost in class > BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated > and will be removed in future versions. Use ClusteringEvaluator instead. You > can also get the cost on the training dataset in the summary. > assert(model.computeCost(dataset) == summary.trainingCost) > Warning:Warning:line (195)method computeCost in class > BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated > and will be removed in future versions. Use ClusteringEvaluator instead. You > can also get the cost on the training dataset in the summary. > model.computeCost(dataset) > > /sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala > Warning:Warning:line (105)Java enum ALLOW_UNQUOTED_CONTROL_CHARS in Java > enum Feature is deprecated: see corresponding Javadoc for more information. > jsonFactory.enable(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS) > /sql/core/src/test/java/test/org/apache/spark/sql/Java8DatasetAggregatorSuite.java > Warning:Warning:line (28)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated > Warning:Warning:line (37)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated > Warning:Warning:line (46)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated > Warning:Warning:line (55)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated > Warning:Warning:line (64)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated > /sql/core/src/test/java/test/org/apache/spark/sql/JavaTestUtils.java > Information:Information:java: > /Users/maxim/proj/eliminate-warning/sql/core/src/test/java/test/org/apache/spark/sql/JavaTestUtils.java > uses unchecked or unsafe operations. > Information:Information:java: Recompile with -Xlint:unchecked for details. > /sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java > Warning:Warning:line (478)java: > json(org.apache.spark.api.java.JavaRDD) in > org.apache.spark.sql.DataFrameReader has been deprecated -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30174) Eliminate warnings :part 4
[
https://issues.apache.org/jira/browse/SPARK-30174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jobit mathew updated SPARK-30174:
-
Description:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
{code:java}
Warning:Warning:line (127)value ENABLE_JOB_SUMMARY in class ParquetOutputFormat
is deprecated: see corresponding Javadoc for more information.
&& conf.get(ParquetOutputFormat.ENABLE_JOB_SUMMARY) == null) {
Warning:Warning:line (261)class ParquetInputSplit in package hadoop is
deprecated: see corresponding Javadoc for more information.
new org.apache.parquet.hadoop.ParquetInputSplit(
Warning:Warning:line (272)method readFooter in class ParquetFileReader is
deprecated: see corresponding Javadoc for more information.
ParquetFileReader.readFooter(sharedConf, filePath,
SKIP_ROW_GROUPS).getFileMetaData
Warning:Warning:line (442)method readFooter in class ParquetFileReader is
deprecated: see corresponding Javadoc for more information.
ParquetFileReader.readFooter(
{code}
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetWriteBuilder.scala
{code:java}
Warning:Warning:line (91)value ENABLE_JOB_SUMMARY in class ParquetOutputFormat
is deprecated: see corresponding Javadoc for more information.
&& conf.get(ParquetOutputFormat.ENABLE_JOB_SUMMARY) == null) {
{code}
was:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetWriteBuilder.scala
> Eliminate warnings :part 4
> --
>
> Key: SPARK-30174
> URL: https://issues.apache.org/jira/browse/SPARK-30174
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: jobit mathew
>Priority: Minor
>
> sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
> {code:java}
> Warning:Warning:line (127)value ENABLE_JOB_SUMMARY in class
> ParquetOutputFormat is deprecated: see corresponding Javadoc for more
> information.
> && conf.get(ParquetOutputFormat.ENABLE_JOB_SUMMARY) == null) {
> Warning:Warning:line (261)class ParquetInputSplit in package hadoop is
> deprecated: see corresponding Javadoc for more information.
> new org.apache.parquet.hadoop.ParquetInputSplit(
> Warning:Warning:line (272)method readFooter in class ParquetFileReader is
> deprecated: see corresponding Javadoc for more information.
> ParquetFileReader.readFooter(sharedConf, filePath,
> SKIP_ROW_GROUPS).getFileMetaData
> Warning:Warning:line (442)method readFooter in class ParquetFileReader is
> deprecated: see corresponding Javadoc for more information.
> ParquetFileReader.readFooter(
> {code}
> sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetWriteBuilder.scala
> {code:java}
> Warning:Warning:line (91)value ENABLE_JOB_SUMMARY in class
> ParquetOutputFormat is deprecated: see corresponding Javadoc for more
> information.
> && conf.get(ParquetOutputFormat.ENABLE_JOB_SUMMARY) == null) {
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30171) Eliminate warnings: part2
[
https://issues.apache.org/jira/browse/SPARK-30171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ABHISHEK KUMAR GUPTA updated SPARK-30171:
-
Summary: Eliminate warnings: part2 (was: Eliminate warnings: )
> Eliminate warnings: part2
> -
>
> Key: SPARK-30171
> URL: https://issues.apache.org/jira/browse/SPARK-30171
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: ABHISHEK KUMAR GUPTA
>Priority: Minor
>
> AvroFunctionsSuite.scala
> Warning:Warning:line (41)method to_avro in package avro is deprecated (since
> 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro' instead.
> val avroDF = df.select(to_avro('id).as("a"), to_avro('str).as("b"))
> Warning:Warning:line (41)method to_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro'
> instead.
> val avroDF = df.select(to_avro('id).as("a"), to_avro('str).as("b"))
> Warning:Warning:line (54)method from_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
> instead.
> checkAnswer(avroDF.select(from_avro('a, avroTypeLong), from_avro('b,
> avroTypeStr)), df)
> Warning:Warning:line (54)method from_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
> instead.
> checkAnswer(avroDF.select(from_avro('a, avroTypeLong), from_avro('b,
> avroTypeStr)), df)
> Warning:Warning:line (59)method to_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro'
> instead.
> val avroStructDF = df.select(to_avro('struct).as("avro"))
> Warning:Warning:line (70)method from_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
> instead.
> checkAnswer(avroStructDF.select(from_avro('avro, avroTypeStruct)), df)
> Warning:Warning:line (76)method to_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro'
> instead.
> val avroStructDF = df.select(to_avro('struct).as("avro"))
> Warning:Warning:line (118)method to_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro'
> instead.
> val readBackOne = dfOne.select(to_avro($"array").as("avro"))
> Warning:Warning:line (119)method from_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
> instead.
> .select(from_avro($"avro", avroTypeArrStruct).as("array"))
> AvroPartitionReaderFactory.scala
> Warning:Warning:line (64)value ignoreExtension in class AvroOptions is
> deprecated (since 3.0): Use the general data source option pathGlobFilter for
> filtering file names
> if (parsedOptions.ignoreExtension ||
> partitionedFile.filePath.endsWith(".avro")) {
> AvroFileFormat.scala
> Warning:Warning:line (98)value ignoreExtension in class AvroOptions is
> deprecated (since 3.0): Use the general data source option pathGlobFilter for
> filtering file names
> if (parsedOptions.ignoreExtension || file.filePath.endsWith(".avro")) {
> AvroUtils.scala
> Warning:Warning:line (55)value ignoreExtension in class AvroOptions is
> deprecated (since 3.0): Use the general data source option pathGlobFilter for
> filtering file names
> inferAvroSchemaFromFiles(files, conf, parsedOptions.ignoreExtension,
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30177) Eliminate warnings: part7
[ https://issues.apache.org/jira/browse/SPARK-30177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ABHISHEK KUMAR GUPTA updated SPARK-30177: - Description: /mllib/src/test/scala/org/apache/spark/ml/clustering/BisectingKMeansSuite.scala Warning:Warning:line (108)method computeCost in class BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated and will be removed in future versions. Use ClusteringEvaluator instead. You can also get the cost on the training dataset in the summary. assert(model.computeCost(dataset) < 0.1) Warning:Warning:line (135)method computeCost in class BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated and will be removed in future versions. Use ClusteringEvaluator instead. You can also get the cost on the training dataset in the summary. assert(model.computeCost(dataset) == summary.trainingCost) Warning:Warning:line (195)method computeCost in class BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated and will be removed in future versions. Use ClusteringEvaluator instead. You can also get the cost on the training dataset in the summary. model.computeCost(dataset) /sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala Warning:Warning:line (105)Java enum ALLOW_UNQUOTED_CONTROL_CHARS in Java enum Feature is deprecated: see corresponding Javadoc for more information. jsonFactory.enable(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS) /sql/core/src/test/java/test/org/apache/spark/sql/Java8DatasetAggregatorSuite.java Warning:Warning:line (28)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated Warning:Warning:line (37)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated Warning:Warning:line (46)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated Warning:Warning:line (55)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated Warning:Warning:line (64)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated /sql/core/src/test/java/test/org/apache/spark/sql/JavaTestUtils.java Information:Information:java: /Users/maxim/proj/eliminate-warning/sql/core/src/test/java/test/org/apache/spark/sql/JavaTestUtils.java uses unchecked or unsafe operations. Information:Information:java: Recompile with -Xlint:unchecked for details. /sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java Warning:Warning:line (478)java: json(org.apache.spark.api.java.JavaRDD) in org.apache.spark.sql.DataFrameReader has been deprecated was: /mllib/src/test/scala/org/apache/spark/ml/clustering/BisectingKMeansSuite.scala Warning:Warning:line (108)method computeCost in class BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated and will be removed in future versions. Use ClusteringEvaluator instead. You can also get the cost on the training dataset in the summary. assert(model.computeCost(dataset) < 0.1) Warning:Warning:line (135)method computeCost in class BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated and will be removed in future versions. Use ClusteringEvaluator instead. You can also get the cost on the training dataset in the summary. assert(model.computeCost(dataset) == summary.trainingCost) Warning:Warning:line (195)method computeCost in class BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated and will be removed in future versions. Use ClusteringEvaluator instead. You can also get the cost on the training dataset in the summary. model.computeCost(dataset) /sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala Warning:Warning:line (105)Java enum ALLOW_UNQUOTED_CONTROL_CHARS in Java enum Feature is deprecated: see corresponding Javadoc for more information. jsonFactory.enable(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS) /sql/core/src/test/java/test/org/apache/spark/sql/Java8DatasetAggregatorSuite.java Warning:Warning:line (28)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated Warning:Warning:line (37)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated Warning:Warning:line (46)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated Warning:Warning:line (55)java: org.apache.spark.sql.expressions.javalang.typed in
[jira] [Created] (SPARK-30178) RobustScaler support bigger numFeatures
zhengruifeng created SPARK-30178:
Summary: RobustScaler support bigger numFeatures
Key: SPARK-30178
URL: https://issues.apache.org/jira/browse/SPARK-30178
Project: Spark
Issue Type: Improvement
Components: ML
Affects Versions: 3.0.0
Reporter: zhengruifeng
It is a bottleneck to collect the whole Array[QuantileSummaries] from executors,
since a QuantileSummaries is a large object, which maintains large arrays of
size
10k({color:#93a6f5}defaultCompressThreshold{color})/50k({color:#93a6f5}defaultHeadSize{color}).
So we need to compute the ranges/medians more distributedly.
In Spark-Shell with default params, I processed dataset with
numFeatures=69,200, and current impl fail due to OOM.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30171) Eliminate warnings:
[
https://issues.apache.org/jira/browse/SPARK-30171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ABHISHEK KUMAR GUPTA updated SPARK-30171:
-
Summary: Eliminate warnings: (was: Eliminate warnings: part 2)
> Eliminate warnings:
>
>
> Key: SPARK-30171
> URL: https://issues.apache.org/jira/browse/SPARK-30171
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: ABHISHEK KUMAR GUPTA
>Priority: Minor
>
> AvroFunctionsSuite.scala
> Warning:Warning:line (41)method to_avro in package avro is deprecated (since
> 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro' instead.
> val avroDF = df.select(to_avro('id).as("a"), to_avro('str).as("b"))
> Warning:Warning:line (41)method to_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro'
> instead.
> val avroDF = df.select(to_avro('id).as("a"), to_avro('str).as("b"))
> Warning:Warning:line (54)method from_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
> instead.
> checkAnswer(avroDF.select(from_avro('a, avroTypeLong), from_avro('b,
> avroTypeStr)), df)
> Warning:Warning:line (54)method from_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
> instead.
> checkAnswer(avroDF.select(from_avro('a, avroTypeLong), from_avro('b,
> avroTypeStr)), df)
> Warning:Warning:line (59)method to_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro'
> instead.
> val avroStructDF = df.select(to_avro('struct).as("avro"))
> Warning:Warning:line (70)method from_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
> instead.
> checkAnswer(avroStructDF.select(from_avro('avro, avroTypeStruct)), df)
> Warning:Warning:line (76)method to_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro'
> instead.
> val avroStructDF = df.select(to_avro('struct).as("avro"))
> Warning:Warning:line (118)method to_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro'
> instead.
> val readBackOne = dfOne.select(to_avro($"array").as("avro"))
> Warning:Warning:line (119)method from_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
> instead.
> .select(from_avro($"avro", avroTypeArrStruct).as("array"))
> AvroPartitionReaderFactory.scala
> Warning:Warning:line (64)value ignoreExtension in class AvroOptions is
> deprecated (since 3.0): Use the general data source option pathGlobFilter for
> filtering file names
> if (parsedOptions.ignoreExtension ||
> partitionedFile.filePath.endsWith(".avro")) {
> AvroFileFormat.scala
> Warning:Warning:line (98)value ignoreExtension in class AvroOptions is
> deprecated (since 3.0): Use the general data source option pathGlobFilter for
> filtering file names
> if (parsedOptions.ignoreExtension || file.filePath.endsWith(".avro")) {
> AvroUtils.scala
> Warning:Warning:line (55)value ignoreExtension in class AvroOptions is
> deprecated (since 3.0): Use the general data source option pathGlobFilter for
> filtering file names
> inferAvroSchemaFromFiles(files, conf, parsedOptions.ignoreExtension,
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-30177) Eliminate warnings: part7
ABHISHEK KUMAR GUPTA created SPARK-30177: Summary: Eliminate warnings: part7 Key: SPARK-30177 URL: https://issues.apache.org/jira/browse/SPARK-30177 Project: Spark Issue Type: Sub-task Components: SQL Affects Versions: 3.0.0 Reporter: ABHISHEK KUMAR GUPTA /mllib/src/test/scala/org/apache/spark/ml/clustering/BisectingKMeansSuite.scala Warning:Warning:line (108)method computeCost in class BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated and will be removed in future versions. Use ClusteringEvaluator instead. You can also get the cost on the training dataset in the summary. assert(model.computeCost(dataset) < 0.1) Warning:Warning:line (135)method computeCost in class BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated and will be removed in future versions. Use ClusteringEvaluator instead. You can also get the cost on the training dataset in the summary. assert(model.computeCost(dataset) == summary.trainingCost) Warning:Warning:line (195)method computeCost in class BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated and will be removed in future versions. Use ClusteringEvaluator instead. You can also get the cost on the training dataset in the summary. model.computeCost(dataset) /sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala Warning:Warning:line (105)Java enum ALLOW_UNQUOTED_CONTROL_CHARS in Java enum Feature is deprecated: see corresponding Javadoc for more information. jsonFactory.enable(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS) /sql/core/src/test/java/test/org/apache/spark/sql/Java8DatasetAggregatorSuite.java Warning:Warning:line (28)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated Warning:Warning:line (37)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated Warning:Warning:line (46)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated Warning:Warning:line (55)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated Warning:Warning:line (64)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated /sql/core/src/test/java/test/org/apache/spark/sql/JavaTestUtils.java Information:Information:java: /Users/maxim/proj/eliminate-warning/sql/core/src/test/java/test/org/apache/spark/sql/JavaTestUtils.java uses unchecked or unsafe operations. Information:Information:java: Recompile with -Xlint:unchecked for details. /sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java Warning:Warning:line (478)java: json(org.apache.spark.api.java.JavaRDD) in org.apache.spark.sql.DataFrameReader has been deprecated -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30176) Eliminate warnings: part 6
[ https://issues.apache.org/jira/browse/SPARK-30176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jobit mathew updated SPARK-30176: - Description: sql/core/src/test/scala/org/apache/spark/sql/DatasetAggregatorSuite.scala sql/core/src/test/scala/org/apache/spark/sql/DatasetBenchmark.scala sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala > Eliminate warnings: part 6 > -- > > Key: SPARK-30176 > URL: https://issues.apache.org/jira/browse/SPARK-30176 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.0.0 >Reporter: jobit mathew >Priority: Minor > > > sql/core/src/test/scala/org/apache/spark/sql/DatasetAggregatorSuite.scala > sql/core/src/test/scala/org/apache/spark/sql/DatasetBenchmark.scala > sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala > sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala > sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala > > sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala > > sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-30176) Eliminate warnings: part 6
jobit mathew created SPARK-30176: Summary: Eliminate warnings: part 6 Key: SPARK-30176 URL: https://issues.apache.org/jira/browse/SPARK-30176 Project: Spark Issue Type: Sub-task Components: SQL Affects Versions: 3.0.0 Reporter: jobit mathew -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30175) Eliminate warnings: part 5
[ https://issues.apache.org/jira/browse/SPARK-30175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991146#comment-16991146 ] Sandeep Katta commented on SPARK-30175: --- thanks for raising, will raise PR soon > Eliminate warnings: part 5 > -- > > Key: SPARK-30175 > URL: https://issues.apache.org/jira/browse/SPARK-30175 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.0.0 >Reporter: jobit mathew >Priority: Minor > > sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/WriteToMicroBatchDataSource.scala > sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala > sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationSuite.scala -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30174) Eliminate warnings :part 4
[ https://issues.apache.org/jira/browse/SPARK-30174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991145#comment-16991145 ] Shivu Sondur commented on SPARK-30174: -- i am working on this > Eliminate warnings :part 4 > -- > > Key: SPARK-30174 > URL: https://issues.apache.org/jira/browse/SPARK-30174 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.0.0 >Reporter: jobit mathew >Priority: Minor > > sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala > sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetWriteBuilder.scala -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30175) Eliminate warnings: part 5
[ https://issues.apache.org/jira/browse/SPARK-30175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jobit mathew updated SPARK-30175: - Description: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/WriteToMicroBatchDataSource.scala sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationSuite.scala > Eliminate warnings: part 5 > -- > > Key: SPARK-30175 > URL: https://issues.apache.org/jira/browse/SPARK-30175 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.0.0 >Reporter: jobit mathew >Priority: Minor > > sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/WriteToMicroBatchDataSource.scala > sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala > sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationSuite.scala -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30175) Eliminate warnings: part 5
[ https://issues.apache.org/jira/browse/SPARK-30175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jobit mathew updated SPARK-30175: - Summary: Eliminate warnings: part 5 (was: Eliminate warnings: part5) > Eliminate warnings: part 5 > -- > > Key: SPARK-30175 > URL: https://issues.apache.org/jira/browse/SPARK-30175 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.0.0 >Reporter: jobit mathew >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-30175) Eliminate warnings: part5
jobit mathew created SPARK-30175: Summary: Eliminate warnings: part5 Key: SPARK-30175 URL: https://issues.apache.org/jira/browse/SPARK-30175 Project: Spark Issue Type: Sub-task Components: SQL Affects Versions: 3.0.0 Reporter: jobit mathew -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30174) Eliminate warnings :part 4
[ https://issues.apache.org/jira/browse/SPARK-30174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jobit mathew updated SPARK-30174: - Description: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetWriteBuilder.scala > Eliminate warnings :part 4 > -- > > Key: SPARK-30174 > URL: https://issues.apache.org/jira/browse/SPARK-30174 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.0.0 >Reporter: jobit mathew >Priority: Minor > > sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala > sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetWriteBuilder.scala -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-30174) Eliminate warnings :part 4
jobit mathew created SPARK-30174: Summary: Eliminate warnings :part 4 Key: SPARK-30174 URL: https://issues.apache.org/jira/browse/SPARK-30174 Project: Spark Issue Type: Sub-task Components: SQL Affects Versions: 3.0.0 Reporter: jobit mathew -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30162) Filter is not being pushed down for Parquet files
[
https://issues.apache.org/jira/browse/SPARK-30162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991137#comment-16991137
]
Aman Omer commented on SPARK-30162:
---
Kindly share the results of spark-shell.
Thanks
> Filter is not being pushed down for Parquet files
> -
>
> Key: SPARK-30162
> URL: https://issues.apache.org/jira/browse/SPARK-30162
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.0
> Environment: pyspark 3.0 preview
> Ubuntu/Centos
> pyarrow 0.14.1
>Reporter: Nasir Ali
>Priority: Major
>
> Filters are not pushed down in Spark 3.0 preview. Also the output of
> "explain" method is different. It is hard to debug in 3.0 whether filters
> were pushed down or not. Below code could reproduce the bug:
>
> {code:java}
> // code placeholder
> df = spark.createDataFrame([("usr1",17.00, "2018-03-10T15:27:18+00:00"),
> ("usr1",13.00, "2018-03-11T12:27:18+00:00"),
> ("usr1",25.00, "2018-03-12T11:27:18+00:00"),
> ("usr1",20.00, "2018-03-13T15:27:18+00:00"),
> ("usr1",17.00, "2018-03-14T12:27:18+00:00"),
> ("usr2",99.00, "2018-03-15T11:27:18+00:00"),
> ("usr2",156.00, "2018-03-22T11:27:18+00:00"),
> ("usr2",17.00, "2018-03-31T11:27:18+00:00"),
> ("usr2",25.00, "2018-03-15T11:27:18+00:00"),
> ("usr2",25.00, "2018-03-16T11:27:18+00:00")
> ],
>["user","id", "ts"])
> df = df.withColumn('ts', df.ts.cast('timestamp'))
> df.write.partitionBy("user").parquet("/home/cnali/data/")df2 =
> spark.read.load("/home/cnali/data/")df2.filter("user=='usr2'").explain(True)
> {code}
> {code:java}
> // Spark 2.4 output
> == Parsed Logical Plan ==
> 'Filter ('user = usr2)
> +- Relation[id#38,ts#39,user#40] parquet== Analyzed Logical Plan ==
> id: double, ts: timestamp, user: string
> Filter (user#40 = usr2)
> +- Relation[id#38,ts#39,user#40] parquet== Optimized Logical Plan ==
> Filter (isnotnull(user#40) && (user#40 = usr2))
> +- Relation[id#38,ts#39,user#40] parquet== Physical Plan ==
> *(1) FileScan parquet [id#38,ts#39,user#40] Batched: true, Format: Parquet,
> Location: InMemoryFileIndex[file:/home/cnali/data], PartitionCount: 1,
> PartitionFilters: [isnotnull(user#40), (user#40 = usr2)], PushedFilters: [],
> ReadSchema: struct{code}
> {code:java}
> // Spark 3.0.0-preview output
> == Parsed Logical Plan ==
> 'Filter ('user = usr2)
> +- RelationV2[id#0, ts#1, user#2] parquet file:/home/cnali/data== Analyzed
> Logical Plan ==
> id: double, ts: timestamp, user: string
> Filter (user#2 = usr2)
> +- RelationV2[id#0, ts#1, user#2] parquet file:/home/cnali/data== Optimized
> Logical Plan ==
> Filter (isnotnull(user#2) AND (user#2 = usr2))
> +- RelationV2[id#0, ts#1, user#2] parquet file:/home/cnali/data== Physical
> Plan ==
> *(1) Project [id#0, ts#1, user#2]
> +- *(1) Filter (isnotnull(user#2) AND (user#2 = usr2))
>+- *(1) ColumnarToRow
> +- BatchScan[id#0, ts#1, user#2] ParquetScan Location:
> InMemoryFileIndex[file:/home/cnali/data], ReadSchema:
> struct
> {code}
> I have tested it on much larger dataset. Spark 3.0 tries to load whole data
> and then apply filter. Whereas Spark 2.4 push down the filter. Above output
> shows that Spark 2.4 applied partition filter but not the Spark 3.0 preview.
>
> Minor: in Spark 3.0 "explain()" output is truncated (maybe fixed length?) and
> it's hard to debug. spark.sql.orc.cache.stripe.details.size=1 doesn't
> work.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-30173) Automatically close stale PRs
Nicholas Chammas created SPARK-30173: Summary: Automatically close stale PRs Key: SPARK-30173 URL: https://issues.apache.org/jira/browse/SPARK-30173 Project: Spark Issue Type: Improvement Components: Project Infra Affects Versions: 3.0.0 Reporter: Nicholas Chammas To manage the number of open PRs we have at any one time, we should automatically close stale PRs with a friendly message. Background discussion: http://apache-spark-developers-list.1001551.n3.nabble.com/Closing-stale-PRs-with-a-GitHub-Action-td28477.html -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30172) Eliminate warnings: part3
[
https://issues.apache.org/jira/browse/SPARK-30172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ABHISHEK KUMAR GUPTA updated SPARK-30172:
-
Description:
/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformationExec.scala
Warning:Warning:line (422)method initialize in class AbstractSerDe is
deprecated: see corresponding Javadoc for more information.
serde.initialize(null, properties)
/sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUDFs.scala
Warning:Warning:line (216)method initialize in class GenericUDTF is
deprecated: see corresponding Javadoc for more information.
protected lazy val outputInspector =
function.initialize(inputInspectors.toArray)
Warning:Warning:line (342)class UDAF in package exec is deprecated: see
corresponding Javadoc for more information.
new GenericUDAFBridge(funcWrapper.createFunction[UDAF]())
Warning:Warning:line (503)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
def serialize(buffer: AggregationBuffer): Array[Byte] = {
Warning:Warning:line (523)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
def deserialize(bytes: Array[Byte]): AggregationBuffer = {
Warning:Warning:line (538)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
case class HiveUDAFBuffer(buf: AggregationBuffer, canDoMerge: Boolean)
Warning:Warning:line (538)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
case class HiveUDAFBuffer(buf: AggregationBuffer, canDoMerge: Boolean)
/sql/hive/src/main/java/org/apache/hadoop/hive/ql/io/orc/SparkOrcNewRecordReader.java
Warning:Warning:line (44)java: getTypes() in org.apache.orc.Reader has been
deprecated
Warning:Warning:line (47)java: getTypes() in org.apache.orc.Reader has been
deprecated
/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
Warning:Warning:line (2,368)method readFooter in class ParquetFileReader is
deprecated: see corresponding Javadoc for more information.
val footer = ParquetFileReader.readFooter(
/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDAFSuite.scala
Warning:Warning:line (202)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
override def getNewAggregationBuffer: AggregationBuffer = new
MockUDAFBuffer(0L, 0L)
Warning:Warning:line (204)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
override def reset(agg: AggregationBuffer): Unit = {
Warning:Warning:line (212)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
override def iterate(agg: AggregationBuffer, parameters: Array[AnyRef]): Unit
= {
Warning:Warning:line (221)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
override def merge(agg: AggregationBuffer, partial: Object): Unit = {
Warning:Warning:line (231)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
override def terminatePartial(agg: AggregationBuffer): AnyRef = {
Warning:Warning:line (236)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
override def terminate(agg: AggregationBuffer): AnyRef = terminatePartial(agg)
Warning:Warning:line (257)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
override def getNewAggregationBuffer: AggregationBuffer = {
Warning:Warning:line (266)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
override def reset(agg: AggregationBuffer): Unit = {
Warning:Warning:line (277)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
override def iterate(agg: AggregationBuffer, parameters: Array[AnyRef]): Unit
= {
Warning:Warning:line (286)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
override def merge(agg: AggregationBuffer, partial: Object): Unit = {
Warning:Warning:line (298)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
override def terminatePartial(agg: AggregationBuffer): AnyRef = {
Warning:Warning:line (310)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see
[jira] [Commented] (SPARK-30172) Eliminate warnings: part3
[
https://issues.apache.org/jira/browse/SPARK-30172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991135#comment-16991135
]
Ankit Raj Boudh commented on SPARK-30172:
-
I will raise PR for this
> Eliminate warnings: part3
> -
>
> Key: SPARK-30172
> URL: https://issues.apache.org/jira/browse/SPARK-30172
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: ABHISHEK KUMAR GUPTA
>Priority: Minor
>
> /sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformationExec.scala
> Warning:Warning:line (422)method initialize in class AbstractSerDe is
> deprecated: see corresponding Javadoc for more information.
> serde.initialize(null, properties)
> /sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUDFs.scala
> Warning:Warning:line (216)method initialize in class GenericUDTF is
> deprecated: see corresponding Javadoc for more information.
> protected lazy val outputInspector =
> function.initialize(inputInspectors.toArray)
> Warning:Warning:line (342)class UDAF in package exec is deprecated: see
> corresponding Javadoc for more information.
> new GenericUDAFBridge(funcWrapper.createFunction[UDAF]())
> Warning:Warning:line (503)trait AggregationBuffer in class
> GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
> information.
> def serialize(buffer: AggregationBuffer): Array[Byte] = {
> Warning:Warning:line (523)trait AggregationBuffer in class
> GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
> information.
> def deserialize(bytes: Array[Byte]): AggregationBuffer = {
> Warning:Warning:line (538)trait AggregationBuffer in class
> GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
> information.
> case class HiveUDAFBuffer(buf: AggregationBuffer, canDoMerge: Boolean)
> Warning:Warning:line (538)trait AggregationBuffer in class
> GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
> information.
> case class HiveUDAFBuffer(buf: AggregationBuffer, canDoMerge: Boolean)
> /sql/hive/src/main/java/org/apache/hadoop/hive/ql/io/orc/SparkOrcNewRecordReader.java
> Warning:Warning:line (44)java: getTypes() in org.apache.orc.Reader has
> been deprecated
> Warning:Warning:line (47)java: getTypes() in org.apache.orc.Reader has
> been deprecated
> /sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
> Warning:Warning:line (2,368)method readFooter in class ParquetFileReader
> is deprecated: see corresponding Javadoc for more information.
> val footer = ParquetFileReader.readFooter(
> /sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDAFSuite.scala
> Warning:Warning:line (202)trait AggregationBuffer in class
> GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
> information.
> override def getNewAggregationBuffer: AggregationBuffer = new
> MockUDAFBuffer(0L, 0L)
> Warning:Warning:line (204)trait AggregationBuffer in class
> GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
> information.
> override def reset(agg: AggregationBuffer): Unit = {
> Warning:Warning:line (212)trait AggregationBuffer in class
> GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
> information.
> override def iterate(agg: AggregationBuffer, parameters: Array[AnyRef]):
> Unit = {
> Warning:Warning:line (221)trait AggregationBuffer in class
> GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
> information.
> override def merge(agg: AggregationBuffer, partial: Object): Unit = {
> Warning:Warning:line (231)trait AggregationBuffer in class
> GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
> information.
> override def terminatePartial(agg: AggregationBuffer): AnyRef = {
> Warning:Warning:line (236)trait AggregationBuffer in class
> GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
> information.
> override def terminate(agg: AggregationBuffer): AnyRef =
> terminatePartial(agg)
> Warning:Warning:line (257)trait AggregationBuffer in class
> GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
> information.
> override def getNewAggregationBuffer: AggregationBuffer = {
> Warning:Warning:line (266)trait AggregationBuffer in class
> GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
> information.
> override def reset(agg: AggregationBuffer): Unit = {
> Warning:Warning:line (277)trait AggregationBuffer in class
> GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
> information.
> override def iterate(agg: AggregationBuffer, parameters: Array[AnyRef]):
> Unit = {
[jira] [Created] (SPARK-30172) Eliminate warnings: part3
ABHISHEK KUMAR GUPTA created SPARK-30172:
Summary: Eliminate warnings: part3
Key: SPARK-30172
URL: https://issues.apache.org/jira/browse/SPARK-30172
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 3.0.0
Reporter: ABHISHEK KUMAR GUPTA
/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformationExec.scala
Warning:Warning:line (422)method initialize in class AbstractSerDe is
deprecated: see corresponding Javadoc for more information.
serde.initialize(null, properties)
/sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUDFs.scala
Warning:Warning:line (216)method initialize in class GenericUDTF is
deprecated: see corresponding Javadoc for more information.
protected lazy val outputInspector =
function.initialize(inputInspectors.toArray)
Warning:Warning:line (342)class UDAF in package exec is deprecated: see
corresponding Javadoc for more information.
new GenericUDAFBridge(funcWrapper.createFunction[UDAF]())
Warning:Warning:line (503)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
def serialize(buffer: AggregationBuffer): Array[Byte] = {
Warning:Warning:line (523)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
def deserialize(bytes: Array[Byte]): AggregationBuffer = {
Warning:Warning:line (538)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
case class HiveUDAFBuffer(buf: AggregationBuffer, canDoMerge: Boolean)
Warning:Warning:line (538)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
case class HiveUDAFBuffer(buf: AggregationBuffer, canDoMerge: Boolean)
/sql/hive/src/main/java/org/apache/hadoop/hive/ql/io/orc/SparkOrcNewRecordReader.java
Warning:Warning:line (44)java: getTypes() in org.apache.orc.Reader has been
deprecated
Warning:Warning:line (47)java: getTypes() in org.apache.orc.Reader has been
deprecated
/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
Warning:Warning:line (2,368)method readFooter in class ParquetFileReader is
deprecated: see corresponding Javadoc for more information.
val footer = ParquetFileReader.readFooter(
/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDAFSuite.scala
Warning:Warning:line (202)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
override def getNewAggregationBuffer: AggregationBuffer = new
MockUDAFBuffer(0L, 0L)
Warning:Warning:line (204)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
override def reset(agg: AggregationBuffer): Unit = {
Warning:Warning:line (212)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
override def iterate(agg: AggregationBuffer, parameters: Array[AnyRef]): Unit
= {
Warning:Warning:line (221)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
override def merge(agg: AggregationBuffer, partial: Object): Unit = {
Warning:Warning:line (231)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
override def terminatePartial(agg: AggregationBuffer): AnyRef = {
Warning:Warning:line (236)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
override def terminate(agg: AggregationBuffer): AnyRef = terminatePartial(agg)
Warning:Warning:line (257)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
override def getNewAggregationBuffer: AggregationBuffer = {
Warning:Warning:line (266)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
override def reset(agg: AggregationBuffer): Unit = {
Warning:Warning:line (277)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
override def iterate(agg: AggregationBuffer, parameters: Array[AnyRef]): Unit
= {
Warning:Warning:line (286)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
override def merge(agg: AggregationBuffer, partial: Object): Unit = {
Warning:Warning:line (298)trait AggregationBuffer in class
GenericUDAFEvaluator is deprecated: see corresponding Javadoc for more
information.
override def
[jira] [Updated] (SPARK-30171) Eliminate warnings: part 2
[
https://issues.apache.org/jira/browse/SPARK-30171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ABHISHEK KUMAR GUPTA updated SPARK-30171:
-
Description:
AvroFunctionsSuite.scala
Warning:Warning:line (41)method to_avro in package avro is deprecated (since
3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro' instead.
val avroDF = df.select(to_avro('id).as("a"), to_avro('str).as("b"))
Warning:Warning:line (41)method to_avro in package avro is deprecated
(since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro' instead.
val avroDF = df.select(to_avro('id).as("a"), to_avro('str).as("b"))
Warning:Warning:line (54)method from_avro in package avro is deprecated
(since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
instead.
checkAnswer(avroDF.select(from_avro('a, avroTypeLong), from_avro('b,
avroTypeStr)), df)
Warning:Warning:line (54)method from_avro in package avro is deprecated
(since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
instead.
checkAnswer(avroDF.select(from_avro('a, avroTypeLong), from_avro('b,
avroTypeStr)), df)
Warning:Warning:line (59)method to_avro in package avro is deprecated
(since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro' instead.
val avroStructDF = df.select(to_avro('struct).as("avro"))
Warning:Warning:line (70)method from_avro in package avro is deprecated
(since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
instead.
checkAnswer(avroStructDF.select(from_avro('avro, avroTypeStruct)), df)
Warning:Warning:line (76)method to_avro in package avro is deprecated
(since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro' instead.
val avroStructDF = df.select(to_avro('struct).as("avro"))
Warning:Warning:line (118)method to_avro in package avro is deprecated
(since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro' instead.
val readBackOne = dfOne.select(to_avro($"array").as("avro"))
Warning:Warning:line (119)method from_avro in package avro is deprecated
(since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
instead.
.select(from_avro($"avro", avroTypeArrStruct).as("array"))
AvroPartitionReaderFactory.scala
Warning:Warning:line (64)value ignoreExtension in class AvroOptions is
deprecated (since 3.0): Use the general data source option pathGlobFilter for
filtering file names
if (parsedOptions.ignoreExtension ||
partitionedFile.filePath.endsWith(".avro")) {
AvroFileFormat.scala
Warning:Warning:line (98)value ignoreExtension in class AvroOptions is
deprecated (since 3.0): Use the general data source option pathGlobFilter for
filtering file names
if (parsedOptions.ignoreExtension || file.filePath.endsWith(".avro")) {
AvroUtils.scala
Warning:Warning:line (55)value ignoreExtension in class AvroOptions is
deprecated (since 3.0): Use the general data source option pathGlobFilter for
filtering file names
inferAvroSchemaFromFiles(files, conf, parsedOptions.ignoreExtension,
was:
AvroFunctionsSuite.scala
Warning:Warning:line (41)method to_avro in package avro is deprecated (since
3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro' instead.
val avroDF = df.select(to_avro('id).as("a"), to_avro('str).as("b"))
Warning:Warning:line (41)method to_avro in package avro is deprecated
(since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro' instead.
val avroDF = df.select(to_avro('id).as("a"), to_avro('str).as("b"))
Warning:Warning:line (54)method from_avro in package avro is deprecated
(since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
instead.
checkAnswer(avroDF.select(from_avro('a, avroTypeLong), from_avro('b,
avroTypeStr)), df)
Warning:Warning:line (54)method from_avro in package avro is deprecated
(since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
instead.
checkAnswer(avroDF.select(from_avro('a, avroTypeLong), from_avro('b,
avroTypeStr)), df)
Warning:Warning:line (59)method to_avro in package avro is deprecated
(since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro' instead.
val avroStructDF = df.select(to_avro('struct).as("avro"))
Warning:Warning:line (70)method from_avro in package avro is deprecated
(since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
instead.
checkAnswer(avroStructDF.select(from_avro('avro, avroTypeStruct)), df)
Warning:Warning:line (76)method to_avro in package avro is deprecated
(since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro' instead.
val avroStructDF = df.select(to_avro('struct).as("avro"))
Warning:Warning:line (118)method to_avro in package avro is deprecated
(since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro'
[jira] [Commented] (SPARK-30171) Eliminate warnings: part 2
[
https://issues.apache.org/jira/browse/SPARK-30171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991134#comment-16991134
]
Ankit Raj Boudh commented on SPARK-30171:
-
i will raise PR for this
> Eliminate warnings: part 2
> --
>
> Key: SPARK-30171
> URL: https://issues.apache.org/jira/browse/SPARK-30171
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: ABHISHEK KUMAR GUPTA
>Priority: Minor
>
> AvroFunctionsSuite.scala
> Warning:Warning:line (41)method to_avro in package avro is deprecated (since
> 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro' instead.
> val avroDF = df.select(to_avro('id).as("a"), to_avro('str).as("b"))
> Warning:Warning:line (41)method to_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro'
> instead.
> val avroDF = df.select(to_avro('id).as("a"), to_avro('str).as("b"))
> Warning:Warning:line (54)method from_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
> instead.
> checkAnswer(avroDF.select(from_avro('a, avroTypeLong), from_avro('b,
> avroTypeStr)), df)
> Warning:Warning:line (54)method from_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
> instead.
> checkAnswer(avroDF.select(from_avro('a, avroTypeLong), from_avro('b,
> avroTypeStr)), df)
> Warning:Warning:line (59)method to_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro'
> instead.
> val avroStructDF = df.select(to_avro('struct).as("avro"))
> Warning:Warning:line (70)method from_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
> instead.
> checkAnswer(avroStructDF.select(from_avro('avro, avroTypeStruct)), df)
> Warning:Warning:line (76)method to_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro'
> instead.
> val avroStructDF = df.select(to_avro('struct).as("avro"))
> Warning:Warning:line (118)method to_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro'
> instead.
> val readBackOne = dfOne.select(to_avro($"array").as("avro"))
> Warning:Warning:line (119)method from_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
> instead.
> .select(from_avro($"avro", avroTypeArrStruct).as("array"))
> /Users/maxim/proj/eliminate-warning/external/avro/src/main/scala/org/apache/spark/sql/v2/avro/AvroPartitionReaderFactory.scala
> Warning:Warning:line (64)value ignoreExtension in class AvroOptions is
> deprecated (since 3.0): Use the general data source option pathGlobFilter for
> filtering file names
> if (parsedOptions.ignoreExtension ||
> partitionedFile.filePath.endsWith(".avro")) {
> /Users/maxim/proj/eliminate-warning/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroFileFormat.scala
> Warning:Warning:line (98)value ignoreExtension in class AvroOptions is
> deprecated (since 3.0): Use the general data source option pathGlobFilter for
> filtering file names
> if (parsedOptions.ignoreExtension || file.filePath.endsWith(".avro")) {
> /Users/maxim/proj/eliminate-warning/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
> Warning:Warning:line (55)value ignoreExtension in class AvroOptions is
> deprecated (since 3.0): Use the general data source option pathGlobFilter for
> filtering file names
> inferAvroSchemaFromFiles(files, conf, parsedOptions.ignoreExtension,
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-30171) Eliminate warnings: part 2
ABHISHEK KUMAR GUPTA created SPARK-30171:
Summary: Eliminate warnings: part 2
Key: SPARK-30171
URL: https://issues.apache.org/jira/browse/SPARK-30171
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 3.0.0
Reporter: ABHISHEK KUMAR GUPTA
AvroFunctionsSuite.scala
Warning:Warning:line (41)method to_avro in package avro is deprecated (since
3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro' instead.
val avroDF = df.select(to_avro('id).as("a"), to_avro('str).as("b"))
Warning:Warning:line (41)method to_avro in package avro is deprecated
(since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro' instead.
val avroDF = df.select(to_avro('id).as("a"), to_avro('str).as("b"))
Warning:Warning:line (54)method from_avro in package avro is deprecated
(since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
instead.
checkAnswer(avroDF.select(from_avro('a, avroTypeLong), from_avro('b,
avroTypeStr)), df)
Warning:Warning:line (54)method from_avro in package avro is deprecated
(since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
instead.
checkAnswer(avroDF.select(from_avro('a, avroTypeLong), from_avro('b,
avroTypeStr)), df)
Warning:Warning:line (59)method to_avro in package avro is deprecated
(since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro' instead.
val avroStructDF = df.select(to_avro('struct).as("avro"))
Warning:Warning:line (70)method from_avro in package avro is deprecated
(since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
instead.
checkAnswer(avroStructDF.select(from_avro('avro, avroTypeStruct)), df)
Warning:Warning:line (76)method to_avro in package avro is deprecated
(since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro' instead.
val avroStructDF = df.select(to_avro('struct).as("avro"))
Warning:Warning:line (118)method to_avro in package avro is deprecated
(since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro' instead.
val readBackOne = dfOne.select(to_avro($"array").as("avro"))
Warning:Warning:line (119)method from_avro in package avro is deprecated
(since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
instead.
.select(from_avro($"avro", avroTypeArrStruct).as("array"))
/Users/maxim/proj/eliminate-warning/external/avro/src/main/scala/org/apache/spark/sql/v2/avro/AvroPartitionReaderFactory.scala
Warning:Warning:line (64)value ignoreExtension in class AvroOptions is
deprecated (since 3.0): Use the general data source option pathGlobFilter for
filtering file names
if (parsedOptions.ignoreExtension ||
partitionedFile.filePath.endsWith(".avro")) {
/Users/maxim/proj/eliminate-warning/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroFileFormat.scala
Warning:Warning:line (98)value ignoreExtension in class AvroOptions is
deprecated (since 3.0): Use the general data source option pathGlobFilter for
filtering file names
if (parsedOptions.ignoreExtension || file.filePath.endsWith(".avro")) {
/Users/maxim/proj/eliminate-warning/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
Warning:Warning:line (55)value ignoreExtension in class AvroOptions is
deprecated (since 3.0): Use the general data source option pathGlobFilter for
filtering file names
inferAvroSchemaFromFiles(files, conf, parsedOptions.ignoreExtension,
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30164) Exclude Hive domain in Unidoc build explicitly
[ https://issues.apache.org/jira/browse/SPARK-30164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-30164: - Component/s: (was: Tests) > Exclude Hive domain in Unidoc build explicitly > -- > > Key: SPARK-30164 > URL: https://issues.apache.org/jira/browse/SPARK-30164 > Project: Spark > Issue Type: Bug > Components: Documentation >Affects Versions: 3.0.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Major > Fix For: 3.0.0 > > > We don't publish this as a part of Spark documentation (see also > https://github.com/apache/spark/blob/master/docs/_plugins/copy_api_dirs.rb#L30) > and most of them are copy of Hive thrift server so that we can officially > use Hive 2.3 release. > It doesn't much make sense to check the documentation generation against > another domain, and that we don't use in documentation publish. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-30164) Exclude Hive domain in Unidoc build explicitly
[ https://issues.apache.org/jira/browse/SPARK-30164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30164. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26800 [https://github.com/apache/spark/pull/26800] > Exclude Hive domain in Unidoc build explicitly > -- > > Key: SPARK-30164 > URL: https://issues.apache.org/jira/browse/SPARK-30164 > Project: Spark > Issue Type: Bug > Components: Documentation, Tests >Affects Versions: 3.0.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Major > Fix For: 3.0.0 > > > We don't publish this as a part of Spark documentation (see also > https://github.com/apache/spark/blob/master/docs/_plugins/copy_api_dirs.rb#L30) > and most of them are copy of Hive thrift server so that we can officially > use Hive 2.3 release. > It doesn't much make sense to check the documentation generation against > another domain, and that we don't use in documentation publish. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30164) Exclude Hive domain in Unidoc build explicitly
[
https://issues.apache.org/jira/browse/SPARK-30164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-30164:
-
Summary: Exclude Hive domain in Unidoc build explicitly (was: Fix document
generation in "sbt unidoc")
> Exclude Hive domain in Unidoc build explicitly
> --
>
> Key: SPARK-30164
> URL: https://issues.apache.org/jira/browse/SPARK-30164
> Project: Spark
> Issue Type: Bug
> Components: Documentation, Tests
>Affects Versions: 3.0.0
>Reporter: Gengliang Wang
>Assignee: Gengliang Wang
>Priority: Major
>
> In the latest master branch, the document generation command:
> {code:java}
> ./build/sbt -Phadoop-2.7 -Phive-2.3 -Pyarn -Phive -Pmesos -Pkinesis-asl
> -Pspark-ganglia-lgpl -Pkubernetes -Phadoop-cloud -Phive-thriftserver unidoc
> {code}
> failed with such message
> {code:java}
> [error]
> /Users/gengliang.wang/Downloads/spark/sql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java:248:
> error: incompatible types:
> org.apache.hive.service.rpc.thrift.TSessionHandle cannot be converted to
> org.apache.hive.service.cli.thrift.TSessionHandle
> [error] resp.setSessionHandle(sessionHandle.toTSessionHandle());
> [error] ^
> [error]
> /Users/gengliang.wang/Downloads/spark/sql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java:259:
> error: incompatible types: org.apache.hive.service.rpc.thrift.TStatus
> cannot be converted to org.apache.hive.service.cli.thrift.TStatus
> [error] resp.setStatus(HiveSQLException.toTStatus(e));
> [error]^
> [error]
> /Users/gengliang.wang/Downloads/spark/sql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java:346:
> error: method getMinVersion in class ThriftCLIService cannot be applied to
> given types;
> [error] TProtocolVersion protocol =
> getMinVersion(CLIService.SERVER_VERSION,
> [error]
> {code}
> To fix it, we should add change "sbt unidoc" to "sbt clean unidoc"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30164) Exclude Hive domain in Unidoc build explicitly
[
https://issues.apache.org/jira/browse/SPARK-30164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-30164:
-
Description:
We don't publish this as a part of Spark documentation (see also
https://github.com/apache/spark/blob/master/docs/_plugins/copy_api_dirs.rb#L30)
and most of them are copy of Hive thrift server so that we can officially use
Hive 2.3 release.
It doesn't much make sense to check the documentation generation against
another domain, and that we don't use in documentation publish.
was:
In the latest master branch, the document generation command:
{code:java}
./build/sbt -Phadoop-2.7 -Phive-2.3 -Pyarn -Phive -Pmesos -Pkinesis-asl
-Pspark-ganglia-lgpl -Pkubernetes -Phadoop-cloud -Phive-thriftserver unidoc
{code}
failed with such message
{code:java}
[error]
/Users/gengliang.wang/Downloads/spark/sql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java:248:
error: incompatible types: org.apache.hive.service.rpc.thrift.TSessionHandle
cannot be converted to org.apache.hive.service.cli.thrift.TSessionHandle
[error] resp.setSessionHandle(sessionHandle.toTSessionHandle());
[error] ^
[error]
/Users/gengliang.wang/Downloads/spark/sql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java:259:
error: incompatible types: org.apache.hive.service.rpc.thrift.TStatus cannot
be converted to org.apache.hive.service.cli.thrift.TStatus
[error] resp.setStatus(HiveSQLException.toTStatus(e));
[error]^
[error]
/Users/gengliang.wang/Downloads/spark/sql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java:346:
error: method getMinVersion in class ThriftCLIService cannot be applied to
given types;
[error] TProtocolVersion protocol = getMinVersion(CLIService.SERVER_VERSION,
[error]
{code}
To fix it, we should add change "sbt unidoc" to "sbt clean unidoc"
> Exclude Hive domain in Unidoc build explicitly
> --
>
> Key: SPARK-30164
> URL: https://issues.apache.org/jira/browse/SPARK-30164
> Project: Spark
> Issue Type: Bug
> Components: Documentation, Tests
>Affects Versions: 3.0.0
>Reporter: Gengliang Wang
>Assignee: Gengliang Wang
>Priority: Major
>
> We don't publish this as a part of Spark documentation (see also
> https://github.com/apache/spark/blob/master/docs/_plugins/copy_api_dirs.rb#L30)
> and most of them are copy of Hive thrift server so that we can officially
> use Hive 2.3 release.
> It doesn't much make sense to check the documentation generation against
> another domain, and that we don't use in documentation publish.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-29922) SHOW FUNCTIONS should look up catalog/table like v2 commands
[ https://issues.apache.org/jira/browse/SPARK-29922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-29922: --- Assignee: Pablo Langa Blanco > SHOW FUNCTIONS should look up catalog/table like v2 commands > > > Key: SPARK-29922 > URL: https://issues.apache.org/jira/browse/SPARK-29922 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.0.0 >Reporter: Pablo Langa Blanco >Assignee: Pablo Langa Blanco >Priority: Major > > SHOW FUNCTIONS should look up catalog/table like v2 commands -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-29922) SHOW FUNCTIONS should look up catalog/table like v2 commands
[ https://issues.apache.org/jira/browse/SPARK-29922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-29922. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26667 [https://github.com/apache/spark/pull/26667] > SHOW FUNCTIONS should look up catalog/table like v2 commands > > > Key: SPARK-29922 > URL: https://issues.apache.org/jira/browse/SPARK-29922 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.0.0 >Reporter: Pablo Langa Blanco >Assignee: Pablo Langa Blanco >Priority: Major > Fix For: 3.0.0 > > > SHOW FUNCTIONS should look up catalog/table like v2 commands -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30168) Eliminate warnings in Parquet datasource
[
https://issues.apache.org/jira/browse/SPARK-30168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991121#comment-16991121
]
Ankit Raj Boudh commented on SPARK-30168:
-
@[~maxgekk], can i start working in this Jira
> Eliminate warnings in Parquet datasource
>
>
> Key: SPARK-30168
> URL: https://issues.apache.org/jira/browse/SPARK-30168
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Maxim Gekk
>Priority: Minor
>
> #
> sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetPartitionReaderFactory.scala
> {code}
> Warning:Warning:line (120)class ParquetInputSplit in package hadoop is
> deprecated: see corresponding Javadoc for more information.
> Option[TimeZone]) => RecordReader[Void, T]): RecordReader[Void, T]
> = {
> Warning:Warning:line (125)class ParquetInputSplit in package hadoop is
> deprecated: see corresponding Javadoc for more information.
> new org.apache.parquet.hadoop.ParquetInputSplit(
> Warning:Warning:line (134)method readFooter in class ParquetFileReader is
> deprecated: see corresponding Javadoc for more information.
> ParquetFileReader.readFooter(conf, filePath,
> SKIP_ROW_GROUPS).getFileMetaData
> Warning:Warning:line (183)class ParquetInputSplit in package hadoop is
> deprecated: see corresponding Javadoc for more information.
> split: ParquetInputSplit,
> Warning:Warning:line (212)class ParquetInputSplit in package hadoop is
> deprecated: see corresponding Javadoc for more information.
> split: ParquetInputSplit,
> {code}
> #
> sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
> {code}
> Warning:Warning:line (55)java: org.apache.parquet.hadoop.ParquetInputSplit in
> org.apache.parquet.hadoop has been deprecated
> Warning:Warning:line (95)java:
> org.apache.parquet.hadoop.ParquetInputSplit in org.apache.parquet.hadoop has
> been deprecated
> Warning:Warning:line (95)java:
> org.apache.parquet.hadoop.ParquetInputSplit in org.apache.parquet.hadoop has
> been deprecated
> Warning:Warning:line (97)java: getRowGroupOffsets() in
> org.apache.parquet.hadoop.ParquetInputSplit has been deprecated
> Warning:Warning:line (105)java:
> readFooter(org.apache.hadoop.conf.Configuration,org.apache.hadoop.fs.Path,org.apache.parquet.format.converter.ParquetMetadataConverter.MetadataFilter)
> in org.apache.parquet.hadoop.ParquetFileReader has been deprecated
> Warning:Warning:line (108)java:
> filterRowGroups(org.apache.parquet.filter2.compat.FilterCompat.Filter,java.util.List,org.apache.parquet.schema.MessageType)
> in org.apache.parquet.filter2.compat.RowGroupFilter has been deprecated
> Warning:Warning:line (111)java:
> readFooter(org.apache.hadoop.conf.Configuration,org.apache.hadoop.fs.Path,org.apache.parquet.format.converter.ParquetMetadataConverter.MetadataFilter)
> in org.apache.parquet.hadoop.ParquetFileReader has been deprecated
> Warning:Warning:line (147)java:
> ParquetFileReader(org.apache.hadoop.conf.Configuration,org.apache.parquet.hadoop.metadata.FileMetaData,org.apache.hadoop.fs.Path,java.util.List,java.util.List)
> in org.apache.parquet.hadoop.ParquetFileReader has been deprecated
> Warning:Warning:line (203)java:
> readFooter(org.apache.hadoop.conf.Configuration,org.apache.hadoop.fs.Path,org.apache.parquet.format.converter.ParquetMetadataConverter.MetadataFilter)
> in org.apache.parquet.hadoop.ParquetFileReader has been deprecated
> Warning:Warning:line (226)java:
> ParquetFileReader(org.apache.hadoop.conf.Configuration,org.apache.parquet.hadoop.metadata.FileMetaData,org.apache.hadoop.fs.Path,java.util.List,java.util.List)
> in org.apache.parquet.hadoop.ParquetFileReader has been deprecated
> {code}
> #
> sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetCompatibilityTest.scala
> #
> sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetInteroperabilitySuite.scala
> #
> sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTest.scala
> #
> sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-20840) Misleading spurious errors when there are Javadoc (Unidoc) breaks
[
https://issues.apache.org/jira/browse/SPARK-20840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-20840:
-
Affects Version/s: 3.0.0
Labels: (was: bulk-closed)
> Misleading spurious errors when there are Javadoc (Unidoc) breaks
> -
>
> Key: SPARK-20840
> URL: https://issues.apache.org/jira/browse/SPARK-20840
> Project: Spark
> Issue Type: Bug
> Components: Build, Project Infra
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> Currently, when there are Javadoc breaks, this seems printing warnings as
> errors.
> For example, the actual errors were as below in
> https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77070/consoleFull
> {code}
> [error]
> /home/jenkins/workspace/SparkPullRequestBuilder@2/core/target/java/org/apache/spark/scheduler/HighlyCompressedMapStatus.java:4:
> error: reference not found
> [error] * than both {@link config.SHUFFLE_ACCURATE_BLOCK_THRESHOLD} and
> [error] ^
> [error]
> /home/jenkins/workspace/SparkPullRequestBuilder@2/core/target/java/org/apache/spark/scheduler/HighlyCompressedMapStatus.java:5:
> error: reference not found
> [error] * {@link config.SHUFFLE_ACCURATE_BLOCK_THRESHOLD_BY_TIMES_AVERAGE} *
> averageSize. It stores the
> [error] ^
> {code}
> but it also prints many errors from generated Java codes as below:
> {code}
> [info] Constructing Javadoc information...
> [error]
> /home/jenkins/workspace/SparkPullRequestBuilder@2/core/target/java/org/apache/spark/scheduler/BlacklistTracker.java:117:
> error: ExecutorAllocationClient is not public in org.apache.spark; cannot be
> accessed from outside package
> [error] public BlacklistTracker
> (org.apache.spark.scheduler.LiveListenerBus listenerBus,
> org.apache.spark.SparkConf conf,
> scala.Option allocationClient,
> org.apache.spark.util.Clock clock) { throw new RuntimeException(); }
> [error]
> ^
> [error]
> /home/jenkins/workspace/SparkPullRequestBuilder@2/core/target/java/org/apache/spark/scheduler/BlacklistTracker.java:118:
> error: ExecutorAllocationClient is not public in org.apache.spark; cannot be
> accessed from outside package
> [error] public BlacklistTracker (org.apache.spark.SparkContext sc,
> scala.Option allocationClient) {
> throw new RuntimeException(); }
> [error]
> ^
> [error]
> /home/jenkins/workspace/SparkPullRequestBuilder@2/core/target/java/org/apache/spark/SparkConf.java:133:
> error: ConfigReader is not public in org.apache.spark.internal.config;
> cannot be accessed from outside package
> [error] private org.apache.spark.internal.config.ConfigReader reader () {
> throw new RuntimeException(); }
> [error]^
> [error]
> /home/jenkins/workspace/SparkPullRequestBuilder@2/core/target/java/org/apache/spark/SparkConf.java:138:
> error: ConfigEntry is not public in org.apache.spark.internal.config; cannot
> be accessed from outside package
> [error] org.apache.spark.SparkConf set
> (org.apache.spark.internal.config.ConfigEntry entry, T value) { throw new
> RuntimeException(); }
> [error]
> ^
> [error]
> /home/jenkins/workspace/SparkPullRequestBuilder@2/core/target/java/org/apache/spark/SparkConf.java:139:
> error: OptionalConfigEntry is not public in
> org.apache.spark.internal.config; cannot be accessed from outside package
> [error] org.apache.spark.SparkConf set
> (org.apache.spark.internal.config.OptionalConfigEntry entry, T value) {
> throw new RuntimeException(); }
> [error]
> ^
> [error]
> /home/jenkins/workspace/SparkPullRequestBuilder@2/core/target/java/org/apache/spark/SparkConf.java:187:
> error: ConfigEntry is not public in org.apache.spark.internal.config; cannot
> be accessed from outside package
> [error] org.apache.spark.SparkConf
> setIfMissing (org.apache.spark.internal.config.ConfigEntry entry, T value)
> { throw new RuntimeException(); }
> [error]
>^
> [error]
> /home/jenkins/workspace/SparkPullRequestBuilder@2/core/target/java/org/apache/spark/SparkConf.java:188:
> error: OptionalConfigEntry is not public in
> org.apache.spark.internal.config; cannot be accessed from outside package
> [error]
[jira] [Reopened] (SPARK-20840) Misleading spurious errors when there are Javadoc (Unidoc) breaks
[
https://issues.apache.org/jira/browse/SPARK-20840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reopened SPARK-20840:
--
> Misleading spurious errors when there are Javadoc (Unidoc) breaks
> -
>
> Key: SPARK-20840
> URL: https://issues.apache.org/jira/browse/SPARK-20840
> Project: Spark
> Issue Type: Bug
> Components: Build, Project Infra
>Affects Versions: 2.2.0
>Reporter: Hyukjin Kwon
>Priority: Major
> Labels: bulk-closed
>
> Currently, when there are Javadoc breaks, this seems printing warnings as
> errors.
> For example, the actual errors were as below in
> https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77070/consoleFull
> {code}
> [error]
> /home/jenkins/workspace/SparkPullRequestBuilder@2/core/target/java/org/apache/spark/scheduler/HighlyCompressedMapStatus.java:4:
> error: reference not found
> [error] * than both {@link config.SHUFFLE_ACCURATE_BLOCK_THRESHOLD} and
> [error] ^
> [error]
> /home/jenkins/workspace/SparkPullRequestBuilder@2/core/target/java/org/apache/spark/scheduler/HighlyCompressedMapStatus.java:5:
> error: reference not found
> [error] * {@link config.SHUFFLE_ACCURATE_BLOCK_THRESHOLD_BY_TIMES_AVERAGE} *
> averageSize. It stores the
> [error] ^
> {code}
> but it also prints many errors from generated Java codes as below:
> {code}
> [info] Constructing Javadoc information...
> [error]
> /home/jenkins/workspace/SparkPullRequestBuilder@2/core/target/java/org/apache/spark/scheduler/BlacklistTracker.java:117:
> error: ExecutorAllocationClient is not public in org.apache.spark; cannot be
> accessed from outside package
> [error] public BlacklistTracker
> (org.apache.spark.scheduler.LiveListenerBus listenerBus,
> org.apache.spark.SparkConf conf,
> scala.Option allocationClient,
> org.apache.spark.util.Clock clock) { throw new RuntimeException(); }
> [error]
> ^
> [error]
> /home/jenkins/workspace/SparkPullRequestBuilder@2/core/target/java/org/apache/spark/scheduler/BlacklistTracker.java:118:
> error: ExecutorAllocationClient is not public in org.apache.spark; cannot be
> accessed from outside package
> [error] public BlacklistTracker (org.apache.spark.SparkContext sc,
> scala.Option allocationClient) {
> throw new RuntimeException(); }
> [error]
> ^
> [error]
> /home/jenkins/workspace/SparkPullRequestBuilder@2/core/target/java/org/apache/spark/SparkConf.java:133:
> error: ConfigReader is not public in org.apache.spark.internal.config;
> cannot be accessed from outside package
> [error] private org.apache.spark.internal.config.ConfigReader reader () {
> throw new RuntimeException(); }
> [error]^
> [error]
> /home/jenkins/workspace/SparkPullRequestBuilder@2/core/target/java/org/apache/spark/SparkConf.java:138:
> error: ConfigEntry is not public in org.apache.spark.internal.config; cannot
> be accessed from outside package
> [error] org.apache.spark.SparkConf set
> (org.apache.spark.internal.config.ConfigEntry entry, T value) { throw new
> RuntimeException(); }
> [error]
> ^
> [error]
> /home/jenkins/workspace/SparkPullRequestBuilder@2/core/target/java/org/apache/spark/SparkConf.java:139:
> error: OptionalConfigEntry is not public in
> org.apache.spark.internal.config; cannot be accessed from outside package
> [error] org.apache.spark.SparkConf set
> (org.apache.spark.internal.config.OptionalConfigEntry entry, T value) {
> throw new RuntimeException(); }
> [error]
> ^
> [error]
> /home/jenkins/workspace/SparkPullRequestBuilder@2/core/target/java/org/apache/spark/SparkConf.java:187:
> error: ConfigEntry is not public in org.apache.spark.internal.config; cannot
> be accessed from outside package
> [error] org.apache.spark.SparkConf
> setIfMissing (org.apache.spark.internal.config.ConfigEntry entry, T value)
> { throw new RuntimeException(); }
> [error]
>^
> [error]
> /home/jenkins/workspace/SparkPullRequestBuilder@2/core/target/java/org/apache/spark/SparkConf.java:188:
> error: OptionalConfigEntry is not public in
> org.apache.spark.internal.config; cannot be accessed from outside package
> [error] org.apache.spark.SparkConf
> setIfMissing
[jira] [Commented] (SPARK-29553) This problemis about using native BLAS to improvement ML/MLLIB performance
[
https://issues.apache.org/jira/browse/SPARK-29553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991098#comment-16991098
]
WuZeyi commented on SPARK-29553:
[~srowen] Sir, I use the cmd line “export OPENBLAS_NUM_THREADS=1” to set env of
OS on the machine where my executor is launched, but it doesn't work either.
I propose a concrete change to the docs, PTAL.
https://github.com/apache/spark/pull/26801
> This problemis about using native BLAS to improvement ML/MLLIB performance
> --
>
> Key: SPARK-29553
> URL: https://issues.apache.org/jira/browse/SPARK-29553
> Project: Spark
> Issue Type: Improvement
> Components: ML, MLlib
>Affects Versions: 2.3.0, 2.4.4
>Reporter: WuZeyi
>Priority: Minor
> Labels: performance
> Attachments: image-2019-11-19-16-11-43-130.png,
> image-2019-11-19-16-13-30-723.png, image-2019-11-21-17-08-15-797.png
>
>
> I use {color:#ff}native BLAS{color} to improvement ML/MLLIB performance
> on Yarn.
> The file {color:#ff}spark-env.sh{color} which is modified by SPARK-21305
> said that I should set {color:#ff}OPENBLAS_NUM_THREADS=1{color} to
> disable multi-threading of OpenBLAS, but it does not take effect.
> I modify {color:#ff}spark.conf{color} to set
> {color:#FF}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color}，and the
> performance improve.
>
>
> I think MKL_NUM_THREADS is the same.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26346) Upgrade parquet to 1.11.0
[ https://issues.apache.org/jira/browse/SPARK-26346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991093#comment-16991093 ] Yuming Wang commented on SPARK-26346: - We are still waiting for the Parquet 1.11.0 final release. > Upgrade parquet to 1.11.0 > - > > Key: SPARK-26346 > URL: https://issues.apache.org/jira/browse/SPARK-26346 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.0.0 >Reporter: Yuming Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30144) MLP param map missing
[
https://issues.apache.org/jira/browse/SPARK-30144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991083#comment-16991083
]
L. C. Hsieh commented on SPARK-30144:
-
Seems so. MultilayerPerceptronParams is missed.
> MLP param map missing
> -
>
> Key: SPARK-30144
> URL: https://issues.apache.org/jira/browse/SPARK-30144
> Project: Spark
> Issue Type: Bug
> Components: MLlib
>Affects Versions: 2.4.4
>Reporter: Glen-Erik Cortes
>Priority: Minor
> Attachments: MLP_params_missing.ipynb,
> data_banknote_authentication.csv
>
>
> Param maps for fitted classifiers are available with all classifiers except
> for the MultilayerPerceptronClassifier.
>
> There is no way to track or know what parameters were best during a
> crossvalidation or which parameters were used for submodels.
>
> {code:java}
> {
> Param(parent='MultilayerPerceptronClassifier_eeab0cc242d1',
> name='featuresCol', doc='features column name'): 'features',
> Param(parent='MultilayerPerceptronClassifier_eeab0cc242d1', name='labelCol',
> doc='label column name'): 'fake_banknote',
> Param(parent='MultilayerPerceptronClassifier_eeab0cc242d1',
> name='predictionCol', doc='prediction column name'): 'prediction',
> Param(parent='MultilayerPerceptronClassifier_eeab0cc242d1',
> name='probabilityCol', doc='Column name for predicted class conditional
> probabilities. Note: Not all models output well-calibrated probability
> estimates! These probabilities should be treated as confidences, not precise
> probabilities'): 'probability',
> Param(parent='MultilayerPerceptronClassifier_eeab0cc242d1',
> name='rawPredictionCol', doc='raw prediction (a.k.a. confidence) column
> name'): 'rawPrediction'}{code}
>
> GBTClassifier for example shows all parameters:
>
> {code:java}
> {
> Param(parent='GBTClassifier_a0e77b3430aa', name='cacheNodeIds', doc='If
> false, the algorithm will pass trees to executors to match instances with
> nodes. If true, the algorithm will cache node IDs for each instance. Caching
> can speed up training of deeper trees.'): False,
> Param(parent='GBTClassifier_a0e77b3430aa', name='checkpointInterval',
> doc='set checkpoint interval (>= 1) or disable checkpoint (-1). E.g. 10 means
> that the cache will get checkpointed every 10 iterations. Note: this setting
> will be ignored if the checkpoint directory is not set in the SparkContext'):
> 10,
> Param(parent='GBTClassifier_a0e77b3430aa', name='featureSubsetStrategy',
> doc='The number of features to consider for splits at each tree node.
> Supported options: auto, all, onethird, sqrt, log2, (0.0-1.0], [1-n].'):
> 'all',
> Param(parent='GBTClassifier_a0e77b3430aa', name='featuresCol', doc='features
> column name'): 'features',
> Param(parent='GBTClassifier_a0e77b3430aa', name='labelCol', doc='label column
> name'): 'fake_banknote', Param(parent='GBTClassifier_a0e77b3430aa',
> name='lossType', doc='Loss function which GBT tries to minimize
> (case-insensitive). Supported options: logistic'): 'logistic',
> Param(parent='GBTClassifier_a0e77b3430aa', name='maxBins', doc='Max number of
> bins for discretizing continuous features. Must be >=2 and >= number of
> categories for any categorical feature.'): 8,
> Param(parent='GBTClassifier_a0e77b3430aa', name='maxDepth', doc='Maximum
> depth of the tree. (>= 0) E.g., depth 0 means 1 leaf node; depth 1 means 1
> internal node + 2 leaf nodes.'): 5,
> Param(parent='GBTClassifier_a0e77b3430aa', name='maxIter', doc='maximum
> number of iterations (>= 0)'): 20,
> Param(parent='GBTClassifier_a0e77b3430aa', name='maxMemoryInMB', doc='Maximum
> memory in MB allocated to histogram aggregation.'): 256,
> Param(parent='GBTClassifier_a0e77b3430aa', name='minInfoGain', doc='Minimum
> information gain for a split to be considered at a tree node.'): 0.0,
> Param(parent='GBTClassifier_a0e77b3430aa', name='minInstancesPerNode',
> doc='Minimum number of instances each child must have after split. If a split
> causes the left or right child to have fewer than minInstancesPerNode, the
> split will be discarded as invalid. Should be >= 1.'): 1,
> Param(parent='GBTClassifier_a0e77b3430aa', name='predictionCol',
> doc='prediction column name'): 'prediction',
> Param(parent='GBTClassifier_a0e77b3430aa', name='seed', doc='random seed'):
> 1234,
> Param(parent='GBTClassifier_a0e77b3430aa', name='stepSize', doc='Step size
> (a.k.a. learning rate) in interval (0, 1] for shrinking the contribution of
> each estimator.'): 0.1,
> Param(parent='GBTClassifier_a0e77b3430aa', name='subsamplingRate',
> doc='Fraction of the training data used for learning each decision tree, in
> range (0, 1].'): 1.0}{code}
>
> See attached ipynb or example notebook here:
> [https://colab.research.google.com/drive/1lwSHioZKlLh96FhGkdYFe6FUuRfTcSxH]
[jira] [Commented] (SPARK-19842) Informational Referential Integrity Constraints Support in Spark
[ https://issues.apache.org/jira/browse/SPARK-19842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991053#comment-16991053 ] Takeshi Yamamuro commented on SPARK-19842: -- Yea, but, there is no active work for this support now. [~smilegator] Is this target 3.0? > Informational Referential Integrity Constraints Support in Spark > > > Key: SPARK-19842 > URL: https://issues.apache.org/jira/browse/SPARK-19842 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.2.0 >Reporter: Ioana Delaney >Priority: Major > Attachments: InformationalRIConstraints.doc > > > *Informational Referential Integrity Constraints Support in Spark* > This work proposes support for _informational primary key_ and _foreign key > (referential integrity) constraints_ in Spark. The main purpose is to open up > an area of query optimization techniques that rely on referential integrity > constraints semantics. > An _informational_ or _statistical constraint_ is a constraint such as a > _unique_, _primary key_, _foreign key_, or _check constraint_, that can be > used by Spark to improve query performance. Informational constraints are not > enforced by the Spark SQL engine; rather, they are used by Catalyst to > optimize the query processing. They provide semantics information that allows > Catalyst to rewrite queries to eliminate joins, push down aggregates, remove > unnecessary Distinct operations, and perform a number of other optimizations. > Informational constraints are primarily targeted to applications that load > and analyze data that originated from a data warehouse. For such > applications, the conditions for a given constraint are known to be true, so > the constraint does not need to be enforced during data load operations. > The attached document covers constraint definition, metastore storage, > constraint validation, and maintenance. The document shows many examples of > query performance improvements that utilize referential integrity constraints > and can be implemented in Spark. > Link to the google doc: > [InformationalRIConstraints|https://docs.google.com/document/d/17r-cOqbKF7Px0xb9L7krKg2-RQB_gD2pxOmklm-ehsw/edit] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-30144) MLP param map missing
[
https://issues.apache.org/jira/browse/SPARK-30144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991050#comment-16991050
]
zhengruifeng edited comment on SPARK-30144 at 12/9/19 1:40 AM:
---
[~huaxingao] It seems that MultilayerPerceptronClassificationModel should
extend MultilayerPerceptronParams to expose the training params.
was (Author: podongfeng):
[~huaxingao] It seems like that MultilayerPerceptronClassificationModel should
extend MultilayerPerceptronParams to expose the training params.
> MLP param map missing
> -
>
> Key: SPARK-30144
> URL: https://issues.apache.org/jira/browse/SPARK-30144
> Project: Spark
> Issue Type: Bug
> Components: MLlib
>Affects Versions: 2.4.4
>Reporter: Glen-Erik Cortes
>Priority: Minor
> Attachments: MLP_params_missing.ipynb,
> data_banknote_authentication.csv
>
>
> Param maps for fitted classifiers are available with all classifiers except
> for the MultilayerPerceptronClassifier.
>
> There is no way to track or know what parameters were best during a
> crossvalidation or which parameters were used for submodels.
>
> {code:java}
> {
> Param(parent='MultilayerPerceptronClassifier_eeab0cc242d1',
> name='featuresCol', doc='features column name'): 'features',
> Param(parent='MultilayerPerceptronClassifier_eeab0cc242d1', name='labelCol',
> doc='label column name'): 'fake_banknote',
> Param(parent='MultilayerPerceptronClassifier_eeab0cc242d1',
> name='predictionCol', doc='prediction column name'): 'prediction',
> Param(parent='MultilayerPerceptronClassifier_eeab0cc242d1',
> name='probabilityCol', doc='Column name for predicted class conditional
> probabilities. Note: Not all models output well-calibrated probability
> estimates! These probabilities should be treated as confidences, not precise
> probabilities'): 'probability',
> Param(parent='MultilayerPerceptronClassifier_eeab0cc242d1',
> name='rawPredictionCol', doc='raw prediction (a.k.a. confidence) column
> name'): 'rawPrediction'}{code}
>
> GBTClassifier for example shows all parameters:
>
> {code:java}
> {
> Param(parent='GBTClassifier_a0e77b3430aa', name='cacheNodeIds', doc='If
> false, the algorithm will pass trees to executors to match instances with
> nodes. If true, the algorithm will cache node IDs for each instance. Caching
> can speed up training of deeper trees.'): False,
> Param(parent='GBTClassifier_a0e77b3430aa', name='checkpointInterval',
> doc='set checkpoint interval (>= 1) or disable checkpoint (-1). E.g. 10 means
> that the cache will get checkpointed every 10 iterations. Note: this setting
> will be ignored if the checkpoint directory is not set in the SparkContext'):
> 10,
> Param(parent='GBTClassifier_a0e77b3430aa', name='featureSubsetStrategy',
> doc='The number of features to consider for splits at each tree node.
> Supported options: auto, all, onethird, sqrt, log2, (0.0-1.0], [1-n].'):
> 'all',
> Param(parent='GBTClassifier_a0e77b3430aa', name='featuresCol', doc='features
> column name'): 'features',
> Param(parent='GBTClassifier_a0e77b3430aa', name='labelCol', doc='label column
> name'): 'fake_banknote', Param(parent='GBTClassifier_a0e77b3430aa',
> name='lossType', doc='Loss function which GBT tries to minimize
> (case-insensitive). Supported options: logistic'): 'logistic',
> Param(parent='GBTClassifier_a0e77b3430aa', name='maxBins', doc='Max number of
> bins for discretizing continuous features. Must be >=2 and >= number of
> categories for any categorical feature.'): 8,
> Param(parent='GBTClassifier_a0e77b3430aa', name='maxDepth', doc='Maximum
> depth of the tree. (>= 0) E.g., depth 0 means 1 leaf node; depth 1 means 1
> internal node + 2 leaf nodes.'): 5,
> Param(parent='GBTClassifier_a0e77b3430aa', name='maxIter', doc='maximum
> number of iterations (>= 0)'): 20,
> Param(parent='GBTClassifier_a0e77b3430aa', name='maxMemoryInMB', doc='Maximum
> memory in MB allocated to histogram aggregation.'): 256,
> Param(parent='GBTClassifier_a0e77b3430aa', name='minInfoGain', doc='Minimum
> information gain for a split to be considered at a tree node.'): 0.0,
> Param(parent='GBTClassifier_a0e77b3430aa', name='minInstancesPerNode',
> doc='Minimum number of instances each child must have after split. If a split
> causes the left or right child to have fewer than minInstancesPerNode, the
> split will be discarded as invalid. Should be >= 1.'): 1,
> Param(parent='GBTClassifier_a0e77b3430aa', name='predictionCol',
> doc='prediction column name'): 'prediction',
> Param(parent='GBTClassifier_a0e77b3430aa', name='seed', doc='random seed'):
> 1234,
> Param(parent='GBTClassifier_a0e77b3430aa', name='stepSize', doc='Step size
> (a.k.a. learning rate) in interval (0, 1] for shrinking the contribution of
> each estimator.'):
[jira] [Commented] (SPARK-30144) MLP param map missing
[
https://issues.apache.org/jira/browse/SPARK-30144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991050#comment-16991050
]
zhengruifeng commented on SPARK-30144:
--
[~huaxingao] It seems like that MultilayerPerceptronClassificationModel should
extend MultilayerPerceptronParams to expose the training params.
> MLP param map missing
> -
>
> Key: SPARK-30144
> URL: https://issues.apache.org/jira/browse/SPARK-30144
> Project: Spark
> Issue Type: Bug
> Components: MLlib
>Affects Versions: 2.4.4
>Reporter: Glen-Erik Cortes
>Priority: Minor
> Attachments: MLP_params_missing.ipynb,
> data_banknote_authentication.csv
>
>
> Param maps for fitted classifiers are available with all classifiers except
> for the MultilayerPerceptronClassifier.
>
> There is no way to track or know what parameters were best during a
> crossvalidation or which parameters were used for submodels.
>
> {code:java}
> {
> Param(parent='MultilayerPerceptronClassifier_eeab0cc242d1',
> name='featuresCol', doc='features column name'): 'features',
> Param(parent='MultilayerPerceptronClassifier_eeab0cc242d1', name='labelCol',
> doc='label column name'): 'fake_banknote',
> Param(parent='MultilayerPerceptronClassifier_eeab0cc242d1',
> name='predictionCol', doc='prediction column name'): 'prediction',
> Param(parent='MultilayerPerceptronClassifier_eeab0cc242d1',
> name='probabilityCol', doc='Column name for predicted class conditional
> probabilities. Note: Not all models output well-calibrated probability
> estimates! These probabilities should be treated as confidences, not precise
> probabilities'): 'probability',
> Param(parent='MultilayerPerceptronClassifier_eeab0cc242d1',
> name='rawPredictionCol', doc='raw prediction (a.k.a. confidence) column
> name'): 'rawPrediction'}{code}
>
> GBTClassifier for example shows all parameters:
>
> {code:java}
> {
> Param(parent='GBTClassifier_a0e77b3430aa', name='cacheNodeIds', doc='If
> false, the algorithm will pass trees to executors to match instances with
> nodes. If true, the algorithm will cache node IDs for each instance. Caching
> can speed up training of deeper trees.'): False,
> Param(parent='GBTClassifier_a0e77b3430aa', name='checkpointInterval',
> doc='set checkpoint interval (>= 1) or disable checkpoint (-1). E.g. 10 means
> that the cache will get checkpointed every 10 iterations. Note: this setting
> will be ignored if the checkpoint directory is not set in the SparkContext'):
> 10,
> Param(parent='GBTClassifier_a0e77b3430aa', name='featureSubsetStrategy',
> doc='The number of features to consider for splits at each tree node.
> Supported options: auto, all, onethird, sqrt, log2, (0.0-1.0], [1-n].'):
> 'all',
> Param(parent='GBTClassifier_a0e77b3430aa', name='featuresCol', doc='features
> column name'): 'features',
> Param(parent='GBTClassifier_a0e77b3430aa', name='labelCol', doc='label column
> name'): 'fake_banknote', Param(parent='GBTClassifier_a0e77b3430aa',
> name='lossType', doc='Loss function which GBT tries to minimize
> (case-insensitive). Supported options: logistic'): 'logistic',
> Param(parent='GBTClassifier_a0e77b3430aa', name='maxBins', doc='Max number of
> bins for discretizing continuous features. Must be >=2 and >= number of
> categories for any categorical feature.'): 8,
> Param(parent='GBTClassifier_a0e77b3430aa', name='maxDepth', doc='Maximum
> depth of the tree. (>= 0) E.g., depth 0 means 1 leaf node; depth 1 means 1
> internal node + 2 leaf nodes.'): 5,
> Param(parent='GBTClassifier_a0e77b3430aa', name='maxIter', doc='maximum
> number of iterations (>= 0)'): 20,
> Param(parent='GBTClassifier_a0e77b3430aa', name='maxMemoryInMB', doc='Maximum
> memory in MB allocated to histogram aggregation.'): 256,
> Param(parent='GBTClassifier_a0e77b3430aa', name='minInfoGain', doc='Minimum
> information gain for a split to be considered at a tree node.'): 0.0,
> Param(parent='GBTClassifier_a0e77b3430aa', name='minInstancesPerNode',
> doc='Minimum number of instances each child must have after split. If a split
> causes the left or right child to have fewer than minInstancesPerNode, the
> split will be discarded as invalid. Should be >= 1.'): 1,
> Param(parent='GBTClassifier_a0e77b3430aa', name='predictionCol',
> doc='prediction column name'): 'prediction',
> Param(parent='GBTClassifier_a0e77b3430aa', name='seed', doc='random seed'):
> 1234,
> Param(parent='GBTClassifier_a0e77b3430aa', name='stepSize', doc='Step size
> (a.k.a. learning rate) in interval (0, 1] for shrinking the contribution of
> each estimator.'): 0.1,
> Param(parent='GBTClassifier_a0e77b3430aa', name='subsamplingRate',
> doc='Fraction of the training data used for learning each decision tree, in
> range (0, 1].'): 1.0}{code}
>
> See attached ipynb or
[jira] [Commented] (SPARK-30169) Eliminate warnings in Kafka connector
[
https://issues.apache.org/jira/browse/SPARK-30169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991048#comment-16991048
]
Aman Omer commented on SPARK-30169:
---
[~maxgekk] If you haven't started, I will take this.
> Eliminate warnings in Kafka connector
> -
>
> Key: SPARK-30169
> URL: https://issues.apache.org/jira/browse/SPARK-30169
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Maxim Gekk
>Priority: Minor
>
> Eliminate compilation warnings in the files:
> {code}
> external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala
> external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/DirectKafkaInputDStream.scala
> external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaDataConsumer.scala
> external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala
> external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
> external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
> external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala
> external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23435) R tests should support latest testthat
[ https://issues.apache.org/jira/browse/SPARK-23435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-23435: - Affects Version/s: 3.0.0 > R tests should support latest testthat > -- > > Key: SPARK-23435 > URL: https://issues.apache.org/jira/browse/SPARK-23435 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 2.3.1, 2.4.0, 3.0.0 >Reporter: Felix Cheung >Assignee: Felix Cheung >Priority: Major > > To follow up on SPARK-22817, the latest version of testthat, 2.0.0 was > released in Dec 2017, and its method has been changed. > In order for our tests to keep working, we need to detect that and call a > different method. > Jenkins is running 1.0.1 though, we need to check if it is going to work. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30144) MLP param map missing
[
https://issues.apache.org/jira/browse/SPARK-30144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991018#comment-16991018
]
Huaxin Gao commented on SPARK-30144:
Currently, MultilayerPerceptronClassificationModel only has params featuresCol,
labelCol, predictionCol, probabilityCol, rawPredictionCol.
[~viirya] [~podongfeng] Are there any reasons why
MultilayerPerceptronClassificationModel doesn't extend
MultilayerPerceptronParams? If not, I will make it extend
MultilayerPerceptronParams.
> MLP param map missing
> -
>
> Key: SPARK-30144
> URL: https://issues.apache.org/jira/browse/SPARK-30144
> Project: Spark
> Issue Type: Bug
> Components: MLlib
>Affects Versions: 2.4.4
>Reporter: Glen-Erik Cortes
>Priority: Minor
> Attachments: MLP_params_missing.ipynb,
> data_banknote_authentication.csv
>
>
> Param maps for fitted classifiers are available with all classifiers except
> for the MultilayerPerceptronClassifier.
>
> There is no way to track or know what parameters were best during a
> crossvalidation or which parameters were used for submodels.
>
> {code:java}
> {
> Param(parent='MultilayerPerceptronClassifier_eeab0cc242d1',
> name='featuresCol', doc='features column name'): 'features',
> Param(parent='MultilayerPerceptronClassifier_eeab0cc242d1', name='labelCol',
> doc='label column name'): 'fake_banknote',
> Param(parent='MultilayerPerceptronClassifier_eeab0cc242d1',
> name='predictionCol', doc='prediction column name'): 'prediction',
> Param(parent='MultilayerPerceptronClassifier_eeab0cc242d1',
> name='probabilityCol', doc='Column name for predicted class conditional
> probabilities. Note: Not all models output well-calibrated probability
> estimates! These probabilities should be treated as confidences, not precise
> probabilities'): 'probability',
> Param(parent='MultilayerPerceptronClassifier_eeab0cc242d1',
> name='rawPredictionCol', doc='raw prediction (a.k.a. confidence) column
> name'): 'rawPrediction'}{code}
>
> GBTClassifier for example shows all parameters:
>
> {code:java}
> {
> Param(parent='GBTClassifier_a0e77b3430aa', name='cacheNodeIds', doc='If
> false, the algorithm will pass trees to executors to match instances with
> nodes. If true, the algorithm will cache node IDs for each instance. Caching
> can speed up training of deeper trees.'): False,
> Param(parent='GBTClassifier_a0e77b3430aa', name='checkpointInterval',
> doc='set checkpoint interval (>= 1) or disable checkpoint (-1). E.g. 10 means
> that the cache will get checkpointed every 10 iterations. Note: this setting
> will be ignored if the checkpoint directory is not set in the SparkContext'):
> 10,
> Param(parent='GBTClassifier_a0e77b3430aa', name='featureSubsetStrategy',
> doc='The number of features to consider for splits at each tree node.
> Supported options: auto, all, onethird, sqrt, log2, (0.0-1.0], [1-n].'):
> 'all',
> Param(parent='GBTClassifier_a0e77b3430aa', name='featuresCol', doc='features
> column name'): 'features',
> Param(parent='GBTClassifier_a0e77b3430aa', name='labelCol', doc='label column
> name'): 'fake_banknote', Param(parent='GBTClassifier_a0e77b3430aa',
> name='lossType', doc='Loss function which GBT tries to minimize
> (case-insensitive). Supported options: logistic'): 'logistic',
> Param(parent='GBTClassifier_a0e77b3430aa', name='maxBins', doc='Max number of
> bins for discretizing continuous features. Must be >=2 and >= number of
> categories for any categorical feature.'): 8,
> Param(parent='GBTClassifier_a0e77b3430aa', name='maxDepth', doc='Maximum
> depth of the tree. (>= 0) E.g., depth 0 means 1 leaf node; depth 1 means 1
> internal node + 2 leaf nodes.'): 5,
> Param(parent='GBTClassifier_a0e77b3430aa', name='maxIter', doc='maximum
> number of iterations (>= 0)'): 20,
> Param(parent='GBTClassifier_a0e77b3430aa', name='maxMemoryInMB', doc='Maximum
> memory in MB allocated to histogram aggregation.'): 256,
> Param(parent='GBTClassifier_a0e77b3430aa', name='minInfoGain', doc='Minimum
> information gain for a split to be considered at a tree node.'): 0.0,
> Param(parent='GBTClassifier_a0e77b3430aa', name='minInstancesPerNode',
> doc='Minimum number of instances each child must have after split. If a split
> causes the left or right child to have fewer than minInstancesPerNode, the
> split will be discarded as invalid. Should be >= 1.'): 1,
> Param(parent='GBTClassifier_a0e77b3430aa', name='predictionCol',
> doc='prediction column name'): 'prediction',
> Param(parent='GBTClassifier_a0e77b3430aa', name='seed', doc='random seed'):
> 1234,
> Param(parent='GBTClassifier_a0e77b3430aa', name='stepSize', doc='Step size
> (a.k.a. learning rate) in interval (0, 1] for shrinking the contribution of
> each estimator.'): 0.1,
>
[jira] [Updated] (SPARK-30170) Eliminate warnings: part 1
[
https://issues.apache.org/jira/browse/SPARK-30170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxim Gekk updated SPARK-30170:
---
Description:
Eliminate compilation warnings in:
# StopWordsRemoverSuite
{code:java}
Warning:Warning:line (245)non-variable type argument String in type pattern
Seq[String] (the underlying of Seq[String]) is unchecked since it is eliminated
by erasure
case Row(r1: Seq[String], e1: Seq[String], r2: Seq[String], e2:
Seq[String]) =>
Warning:Warning:line (245)non-variable type argument String in type pattern
Seq[String] (the underlying of Seq[String]) is unchecked since it is eliminated
by erasure
case Row(r1: Seq[String], e1: Seq[String], r2: Seq[String], e2:
Seq[String]) =>
Warning:Warning:line (245)non-variable type argument String in type pattern
Seq[String] (the underlying of Seq[String]) is unchecked since it is eliminated
by erasure
case Row(r1: Seq[String], e1: Seq[String], r2: Seq[String], e2:
Seq[String]) =>
Warning:Warning:line (245)non-variable type argument String in type pattern
Seq[String] (the underlying of Seq[String]) is unchecked since it is eliminated
by erasure
case Row(r1: Seq[String], e1: Seq[String], r2: Seq[String], e2:
Seq[String]) =>
Warning:Warning:line (271)non-variable type argument String in type pattern
Seq[String] (the underlying of Seq[String]) is unchecked since it is eliminated
by erasure
case Row(r1: Seq[String], e1: Seq[String], r2: Seq[String], e2:
Seq[String]) =>
Warning:Warning:line (271)non-variable type argument String in type pattern
Seq[String] (the underlying of Seq[String]) is unchecked since it is eliminated
by erasure
case Row(r1: Seq[String], e1: Seq[String], r2: Seq[String], e2:
Seq[String]) =>
Warning:Warning:line (271)non-variable type argument String in type pattern
Seq[String] (the underlying of Seq[String]) is unchecked since it is eliminated
by erasure
case Row(r1: Seq[String], e1: Seq[String], r2: Seq[String], e2:
Seq[String]) =>
Warning:Warning:line (271)non-variable type argument String in type pattern
Seq[String] (the underlying of Seq[String]) is unchecked since it is eliminated
by erasure
case Row(r1: Seq[String], e1: Seq[String], r2: Seq[String], e2:
Seq[String]) =>
{code}
# MLTest.scala
{code:java}
Warning:Warning:line (88)match may not be exhaustive.
It would fail on the following inputs: NumericAttribute(), UnresolvedAttribute
val n = Attribute.fromStructField(dataframe.schema(colName)) match {
{code}
# FloatType.scala
{code:java}
Warning:Warning:line (81)method apply in object BigDecimal is deprecated (since
2.11.0): The default conversion from Float may not do what you want. Use
BigDecimal.decimal for a String representation, or explicitly convert the Float
with .toDouble.
def quot(x: Float, y: Float): Float = (BigDecimal(x) quot
BigDecimal(y)).floatValue
Warning:Warning:line (81)method apply in object BigDecimal is deprecated
(since 2.11.0): The default conversion from Float may not do what you want. Use
BigDecimal.decimal for a String representation, or explicitly convert the Float
with .toDouble.
def quot(x: Float, y: Float): Float = (BigDecimal(x) quot
BigDecimal(y)).floatValue
Warning:Warning:line (82)method apply in object BigDecimal is deprecated
(since 2.11.0): The default conversion from Float may not do what you want. Use
BigDecimal.decimal for a String representation, or explicitly convert the Float
with .toDouble.
def rem(x: Float, y: Float): Float = (BigDecimal(x) remainder
BigDecimal(y)).floatValue
Warning:Warning:line (82)method apply in object BigDecimal is deprecated
(since 2.11.0): The default conversion from Float may not do what you want. Use
BigDecimal.decimal for a String representation, or explicitly convert the Float
with .toDouble.
def rem(x: Float, y: Float): Float = (BigDecimal(x) remainder
BigDecimal(y)).floatValue
{code}
# AnalysisExternalCatalogSuite.scala
{code:java}
Warning:Warning:line (62)method verifyZeroInteractions in class Mockito is
deprecated: see corresponding Javadoc for more information.
verifyZeroInteractions(catalog)
{code}
# CSVExprUtilsSuite.scala
{code:java}
Warning:Warning:line (81)Octal escape literals are deprecated, use \u
instead.
("\0", Some("\u"), None)
{code}
# CollectionExpressionsSuite.scala, HashExpressionsSuite.scala,
ExpressionParserSuite.scala
{code:java}
Warning:Warning:line (39)implicit conversion method stringToUTF8Str should be
enabled
by making the implicit value scala.language.implicitConversions visible.
This can be achieved by adding the import clause 'import
scala.language.implicitConversions'
or by setting the compiler option -language:implicitConversions.
See the Scaladoc for value scala.language.implicitConversions for a discussion
why the feature should be explicitly enabled.
[jira] [Commented] (SPARK-30170) Eliminate warnings: part 1
[
https://issues.apache.org/jira/browse/SPARK-30170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16990989#comment-16990989
]
Maxim Gekk commented on SPARK-30170:
I am working on this
> Eliminate warnings: part 1
> --
>
> Key: SPARK-30170
> URL: https://issues.apache.org/jira/browse/SPARK-30170
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Maxim Gekk
>Priority: Minor
>
> Eliminate compilation warnings in:
> # StopWordsRemoverSuite
> {code}
> Warning:Warning:line (245)non-variable type argument String in type pattern
> Seq[String] (the underlying of Seq[String]) is unchecked since it is
> eliminated by erasure
> case Row(r1: Seq[String], e1: Seq[String], r2: Seq[String], e2:
> Seq[String]) =>
> Warning:Warning:line (245)non-variable type argument String in type
> pattern Seq[String] (the underlying of Seq[String]) is unchecked since it is
> eliminated by erasure
> case Row(r1: Seq[String], e1: Seq[String], r2: Seq[String], e2:
> Seq[String]) =>
> Warning:Warning:line (245)non-variable type argument String in type
> pattern Seq[String] (the underlying of Seq[String]) is unchecked since it is
> eliminated by erasure
> case Row(r1: Seq[String], e1: Seq[String], r2: Seq[String], e2:
> Seq[String]) =>
> Warning:Warning:line (245)non-variable type argument String in type
> pattern Seq[String] (the underlying of Seq[String]) is unchecked since it is
> eliminated by erasure
> case Row(r1: Seq[String], e1: Seq[String], r2: Seq[String], e2:
> Seq[String]) =>
> Warning:Warning:line (271)non-variable type argument String in type
> pattern Seq[String] (the underlying of Seq[String]) is unchecked since it is
> eliminated by erasure
> case Row(r1: Seq[String], e1: Seq[String], r2: Seq[String], e2:
> Seq[String]) =>
> Warning:Warning:line (271)non-variable type argument String in type
> pattern Seq[String] (the underlying of Seq[String]) is unchecked since it is
> eliminated by erasure
> case Row(r1: Seq[String], e1: Seq[String], r2: Seq[String], e2:
> Seq[String]) =>
> Warning:Warning:line (271)non-variable type argument String in type
> pattern Seq[String] (the underlying of Seq[String]) is unchecked since it is
> eliminated by erasure
> case Row(r1: Seq[String], e1: Seq[String], r2: Seq[String], e2:
> Seq[String]) =>
> Warning:Warning:line (271)non-variable type argument String in type
> pattern Seq[String] (the underlying of Seq[String]) is unchecked since it is
> eliminated by erasure
> case Row(r1: Seq[String], e1: Seq[String], r2: Seq[String], e2:
> Seq[String]) =>
> {code}
> # MLTest.scala
> {code}
> Warning:Warning:line (88)match may not be exhaustive.
> It would fail on the following inputs: NumericAttribute(), UnresolvedAttribute
> val n = Attribute.fromStructField(dataframe.schema(colName)) match {
> {code}
> # FloatType.scala
> {code}
> Warning:Warning:line (81)method apply in object BigDecimal is deprecated
> (since 2.11.0): The default conversion from Float may not do what you want.
> Use BigDecimal.decimal for a String representation, or explicitly convert the
> Float with .toDouble.
> def quot(x: Float, y: Float): Float = (BigDecimal(x) quot
> BigDecimal(y)).floatValue
> Warning:Warning:line (81)method apply in object BigDecimal is deprecated
> (since 2.11.0): The default conversion from Float may not do what you want.
> Use BigDecimal.decimal for a String representation, or explicitly convert the
> Float with .toDouble.
> def quot(x: Float, y: Float): Float = (BigDecimal(x) quot
> BigDecimal(y)).floatValue
> Warning:Warning:line (82)method apply in object BigDecimal is deprecated
> (since 2.11.0): The default conversion from Float may not do what you want.
> Use BigDecimal.decimal for a String representation, or explicitly convert the
> Float with .toDouble.
> def rem(x: Float, y: Float): Float = (BigDecimal(x) remainder
> BigDecimal(y)).floatValue
> Warning:Warning:line (82)method apply in object BigDecimal is deprecated
> (since 2.11.0): The default conversion from Float may not do what you want.
> Use BigDecimal.decimal for a String representation, or explicitly convert the
> Float with .toDouble.
> def rem(x: Float, y: Float): Float = (BigDecimal(x) remainder
> BigDecimal(y)).floatValue
> {code}
> # AnalysisExternalCatalogSuite.scala
> {code}
> Warning:Warning:line (62)method verifyZeroInteractions in class Mockito is
> deprecated: see corresponding Javadoc for more information.
> verifyZeroInteractions(catalog)
> {code}
> # CSVExprUtilsSuite.scala
> {code}
> Warning:Warning:line (81)Octal escape literals are deprecated, use \u
> instead.
> ("\0", Some("\u"), None)
> {code}
> #
[jira] [Created] (SPARK-30170) Eliminate warnings: part 1
Maxim Gekk created SPARK-30170:
--
Summary: Eliminate warnings: part 1
Key: SPARK-30170
URL: https://issues.apache.org/jira/browse/SPARK-30170
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 3.0.0
Reporter: Maxim Gekk
Eliminate compilation warnings in:
# StopWordsRemoverSuite
{code}
Warning:Warning:line (245)non-variable type argument String in type pattern
Seq[String] (the underlying of Seq[String]) is unchecked since it is eliminated
by erasure
case Row(r1: Seq[String], e1: Seq[String], r2: Seq[String], e2:
Seq[String]) =>
Warning:Warning:line (245)non-variable type argument String in type pattern
Seq[String] (the underlying of Seq[String]) is unchecked since it is eliminated
by erasure
case Row(r1: Seq[String], e1: Seq[String], r2: Seq[String], e2:
Seq[String]) =>
Warning:Warning:line (245)non-variable type argument String in type pattern
Seq[String] (the underlying of Seq[String]) is unchecked since it is eliminated
by erasure
case Row(r1: Seq[String], e1: Seq[String], r2: Seq[String], e2:
Seq[String]) =>
Warning:Warning:line (245)non-variable type argument String in type pattern
Seq[String] (the underlying of Seq[String]) is unchecked since it is eliminated
by erasure
case Row(r1: Seq[String], e1: Seq[String], r2: Seq[String], e2:
Seq[String]) =>
Warning:Warning:line (271)non-variable type argument String in type pattern
Seq[String] (the underlying of Seq[String]) is unchecked since it is eliminated
by erasure
case Row(r1: Seq[String], e1: Seq[String], r2: Seq[String], e2:
Seq[String]) =>
Warning:Warning:line (271)non-variable type argument String in type pattern
Seq[String] (the underlying of Seq[String]) is unchecked since it is eliminated
by erasure
case Row(r1: Seq[String], e1: Seq[String], r2: Seq[String], e2:
Seq[String]) =>
Warning:Warning:line (271)non-variable type argument String in type pattern
Seq[String] (the underlying of Seq[String]) is unchecked since it is eliminated
by erasure
case Row(r1: Seq[String], e1: Seq[String], r2: Seq[String], e2:
Seq[String]) =>
Warning:Warning:line (271)non-variable type argument String in type pattern
Seq[String] (the underlying of Seq[String]) is unchecked since it is eliminated
by erasure
case Row(r1: Seq[String], e1: Seq[String], r2: Seq[String], e2:
Seq[String]) =>
{code}
# MLTest.scala
{code}
Warning:Warning:line (88)match may not be exhaustive.
It would fail on the following inputs: NumericAttribute(), UnresolvedAttribute
val n = Attribute.fromStructField(dataframe.schema(colName)) match {
{code}
# FloatType.scala
{code}
Warning:Warning:line (81)method apply in object BigDecimal is deprecated (since
2.11.0): The default conversion from Float may not do what you want. Use
BigDecimal.decimal for a String representation, or explicitly convert the Float
with .toDouble.
def quot(x: Float, y: Float): Float = (BigDecimal(x) quot
BigDecimal(y)).floatValue
Warning:Warning:line (81)method apply in object BigDecimal is deprecated
(since 2.11.0): The default conversion from Float may not do what you want. Use
BigDecimal.decimal for a String representation, or explicitly convert the Float
with .toDouble.
def quot(x: Float, y: Float): Float = (BigDecimal(x) quot
BigDecimal(y)).floatValue
Warning:Warning:line (82)method apply in object BigDecimal is deprecated
(since 2.11.0): The default conversion from Float may not do what you want. Use
BigDecimal.decimal for a String representation, or explicitly convert the Float
with .toDouble.
def rem(x: Float, y: Float): Float = (BigDecimal(x) remainder
BigDecimal(y)).floatValue
Warning:Warning:line (82)method apply in object BigDecimal is deprecated
(since 2.11.0): The default conversion from Float may not do what you want. Use
BigDecimal.decimal for a String representation, or explicitly convert the Float
with .toDouble.
def rem(x: Float, y: Float): Float = (BigDecimal(x) remainder
BigDecimal(y)).floatValue
{code}
# AnalysisExternalCatalogSuite.scala
{code}
Warning:Warning:line (62)method verifyZeroInteractions in class Mockito is
deprecated: see corresponding Javadoc for more information.
verifyZeroInteractions(catalog)
{code}
# CSVExprUtilsSuite.scala
{code}
Warning:Warning:line (81)Octal escape literals are deprecated, use \u
instead.
("\0", Some("\u"), None)
{code}
# CollectionExpressionsSuite.scala, ashExpressionsSuite.scala,
ExpressionParserSuite.scala
{code}
Warning:Warning:line (39)implicit conversion method stringToUTF8Str should be
enabled
by making the implicit value scala.language.implicitConversions visible.
This can be achieved by adding the import clause 'import
scala.language.implicitConversions'
or by setting the compiler option -language:implicitConversions.
See
[jira] [Created] (SPARK-30169) Eliminate warnings in Kafka connector
Maxim Gekk created SPARK-30169:
--
Summary: Eliminate warnings in Kafka connector
Key: SPARK-30169
URL: https://issues.apache.org/jira/browse/SPARK-30169
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 3.0.0
Reporter: Maxim Gekk
Eliminate compilation warnings in the files:
{code}
external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala
external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/DirectKafkaInputDStream.scala
external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaDataConsumer.scala
external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala
external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30166) Eliminate warnings in JSONOptions
[
https://issues.apache.org/jira/browse/SPARK-30166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxim Gekk updated SPARK-30166:
---
Summary: Eliminate warnings in JSONOptions (was: Eliminate compilation
warnings in JSONOptions)
> Eliminate warnings in JSONOptions
> -
>
> Key: SPARK-30166
> URL: https://issues.apache.org/jira/browse/SPARK-30166
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Maxim Gekk
>Priority: Minor
>
> Scala 2.12 outputs the following warnings for JSONOptions:
> {code}
> sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala
> Warning:Warning:line (137)Java enum ALLOW_NUMERIC_LEADING_ZEROS in Java
> enum Feature is deprecated: see corresponding Javadoc for more information.
> factory.configure(JsonParser.Feature.ALLOW_NUMERIC_LEADING_ZEROS,
> allowNumericLeadingZeros)
> Warning:Warning:line (138)Java enum ALLOW_NON_NUMERIC_NUMBERS in Java
> enum Feature is deprecated: see corresponding Javadoc for more information.
> factory.configure(JsonParser.Feature.ALLOW_NON_NUMERIC_NUMBERS,
> allowNonNumericNumbers)
> Warning:Warning:line (139)Java enum
> ALLOW_BACKSLASH_ESCAPING_ANY_CHARACTER in Java enum Feature is deprecated:
> see corresponding Javadoc for more information.
>
> factory.configure(JsonParser.Feature.ALLOW_BACKSLASH_ESCAPING_ANY_CHARACTER,
> Warning:Warning:line (141)Java enum ALLOW_UNQUOTED_CONTROL_CHARS in Java
> enum Feature is deprecated: see corresponding Javadoc for more information.
> factory.configure(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS,
> allowUnquotedControlChars)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-30168) Eliminate warnings in Parquet datasource
Maxim Gekk created SPARK-30168:
--
Summary: Eliminate warnings in Parquet datasource
Key: SPARK-30168
URL: https://issues.apache.org/jira/browse/SPARK-30168
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 3.0.0
Reporter: Maxim Gekk
#
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetPartitionReaderFactory.scala
{code}
Warning:Warning:line (120)class ParquetInputSplit in package hadoop is
deprecated: see corresponding Javadoc for more information.
Option[TimeZone]) => RecordReader[Void, T]): RecordReader[Void, T] = {
Warning:Warning:line (125)class ParquetInputSplit in package hadoop is
deprecated: see corresponding Javadoc for more information.
new org.apache.parquet.hadoop.ParquetInputSplit(
Warning:Warning:line (134)method readFooter in class ParquetFileReader is
deprecated: see corresponding Javadoc for more information.
ParquetFileReader.readFooter(conf, filePath,
SKIP_ROW_GROUPS).getFileMetaData
Warning:Warning:line (183)class ParquetInputSplit in package hadoop is
deprecated: see corresponding Javadoc for more information.
split: ParquetInputSplit,
Warning:Warning:line (212)class ParquetInputSplit in package hadoop is
deprecated: see corresponding Javadoc for more information.
split: ParquetInputSplit,
{code}
#
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
{code}
Warning:Warning:line (55)java: org.apache.parquet.hadoop.ParquetInputSplit in
org.apache.parquet.hadoop has been deprecated
Warning:Warning:line (95)java: org.apache.parquet.hadoop.ParquetInputSplit
in org.apache.parquet.hadoop has been deprecated
Warning:Warning:line (95)java: org.apache.parquet.hadoop.ParquetInputSplit
in org.apache.parquet.hadoop has been deprecated
Warning:Warning:line (97)java: getRowGroupOffsets() in
org.apache.parquet.hadoop.ParquetInputSplit has been deprecated
Warning:Warning:line (105)java:
readFooter(org.apache.hadoop.conf.Configuration,org.apache.hadoop.fs.Path,org.apache.parquet.format.converter.ParquetMetadataConverter.MetadataFilter)
in org.apache.parquet.hadoop.ParquetFileReader has been deprecated
Warning:Warning:line (108)java:
filterRowGroups(org.apache.parquet.filter2.compat.FilterCompat.Filter,java.util.List,org.apache.parquet.schema.MessageType)
in org.apache.parquet.filter2.compat.RowGroupFilter has been deprecated
Warning:Warning:line (111)java:
readFooter(org.apache.hadoop.conf.Configuration,org.apache.hadoop.fs.Path,org.apache.parquet.format.converter.ParquetMetadataConverter.MetadataFilter)
in org.apache.parquet.hadoop.ParquetFileReader has been deprecated
Warning:Warning:line (147)java:
ParquetFileReader(org.apache.hadoop.conf.Configuration,org.apache.parquet.hadoop.metadata.FileMetaData,org.apache.hadoop.fs.Path,java.util.List,java.util.List)
in org.apache.parquet.hadoop.ParquetFileReader has been deprecated
Warning:Warning:line (203)java:
readFooter(org.apache.hadoop.conf.Configuration,org.apache.hadoop.fs.Path,org.apache.parquet.format.converter.ParquetMetadataConverter.MetadataFilter)
in org.apache.parquet.hadoop.ParquetFileReader has been deprecated
Warning:Warning:line (226)java:
ParquetFileReader(org.apache.hadoop.conf.Configuration,org.apache.parquet.hadoop.metadata.FileMetaData,org.apache.hadoop.fs.Path,java.util.List,java.util.List)
in org.apache.parquet.hadoop.ParquetFileReader has been deprecated
{code}
#
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetCompatibilityTest.scala
#
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetInteroperabilitySuite.scala
#
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTest.scala
# sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26346) Upgrade parquet to 1.11.0
[ https://issues.apache.org/jira/browse/SPARK-26346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16990957#comment-16990957 ] Dongjoon Hyun commented on SPARK-26346: --- Please make a PR, [~toopt4]. Then, we will review. > Upgrade parquet to 1.11.0 > - > > Key: SPARK-26346 > URL: https://issues.apache.org/jira/browse/SPARK-26346 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.0.0 >Reporter: Yuming Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28264) Revisiting Python / pandas UDF
[ https://issues.apache.org/jira/browse/SPARK-28264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16990958#comment-16990958 ] Reynold Xin commented on SPARK-28264: - Sounds good. Thanks for doing this [~hyukjin.kwon]! > Revisiting Python / pandas UDF > -- > > Key: SPARK-28264 > URL: https://issues.apache.org/jira/browse/SPARK-28264 > Project: Spark > Issue Type: Improvement > Components: PySpark, SQL >Affects Versions: 3.0.0 >Reporter: Reynold Xin >Assignee: Reynold Xin >Priority: Critical > > In the past two years, the pandas UDFs are perhaps the most important changes > to Spark for Python data science. However, these functionalities have evolved > organically, leading to some inconsistencies and confusions among users. This > document revisits UDF definition and naming, as a result of discussions among > Xiangrui, Li Jin, Hyukjin, and Reynold. > > See document here: > [https://docs.google.com/document/d/10Pkl-rqygGao2xQf6sddt0b-4FYK4g8qr_bXLKTL65A/edit#|https://docs.google.com/document/d/10Pkl-rqygGao2xQf6sddt0b-4FYK4g8qr_bXLKTL65A/edit] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30165) Eliminate compilation warnings
[ https://issues.apache.org/jira/browse/SPARK-30165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16990925#comment-16990925 ] Maxim Gekk commented on SPARK-30165: [~aman_omer] Feel free to take a sub-set of warnings and create a sub-task to fix them. > Eliminate compilation warnings > -- > > Key: SPARK-30165 > URL: https://issues.apache.org/jira/browse/SPARK-30165 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Maxim Gekk >Priority: Minor > Attachments: spark_warnings.txt > > > This is an umbrella ticket for sub-tasks for eliminating compilation > warnings. I dumped all warnings to the spark_warnings.txt file attached to > the ticket. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12606) Scala/Java compatibility issue Re: how to extend java transformer from Scala UnaryTransformer ?
[
https://issues.apache.org/jira/browse/SPARK-12606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16990892#comment-16990892
]
Sabarish Sasidharan commented on SPARK-12606:
-
For folks who run into this, I was able to solve this issue by using a static
uid generated like below
{code:java}
private static final String uid =
Identifiable$.MODULE$.randomUID("sentence_embed_finisher_2");
@Override
public String uid() {
return uid;
}
{code}
> Scala/Java compatibility issue Re: how to extend java transformer from Scala
> UnaryTransformer ?
> ---
>
> Key: SPARK-12606
> URL: https://issues.apache.org/jira/browse/SPARK-12606
> Project: Spark
> Issue Type: Bug
> Components: ML
>Affects Versions: 1.5.2
> Environment: Java 8, Mac OS, Spark-1.5.2
>Reporter: Andrew Davidson
>Priority: Major
> Labels: transformers
>
> Hi Andy,
> I suspect that you hit the Scala/Java compatibility issue, I can also
> reproduce this issue, so could you file a JIRA to track this issue?
> Yanbo
> 2016-01-02 3:38 GMT+08:00 Andy Davidson :
> I am trying to write a trivial transformer I use use in my pipeline. I am
> using java and spark 1.5.2. It was suggested that I use the Tokenize.scala
> class as an example. This should be very easy how ever I do not understand
> Scala, I am having trouble debugging the following exception.
> Any help would be greatly appreciated.
> Happy New Year
> Andy
> java.lang.IllegalArgumentException: requirement failed: Param null__inputCol
> does not belong to Stemmer_2f3aa96d-7919-4eaa-ad54-f7c620b92d1c.
> at scala.Predef$.require(Predef.scala:233)
> at org.apache.spark.ml.param.Params$class.shouldOwn(params.scala:557)
> at org.apache.spark.ml.param.Params$class.set(params.scala:436)
> at org.apache.spark.ml.PipelineStage.set(Pipeline.scala:37)
> at org.apache.spark.ml.param.Params$class.set(params.scala:422)
> at org.apache.spark.ml.PipelineStage.set(Pipeline.scala:37)
> at
> org.apache.spark.ml.UnaryTransformer.setInputCol(Transformer.scala:83)
> at com.pws.xxx.ml.StemmerTest.test(StemmerTest.java:30)
> public class StemmerTest extends AbstractSparkTest {
> @Test
> public void test() {
> Stemmer stemmer = new Stemmer()
> .setInputCol("raw”) //line 30
> .setOutputCol("filtered");
> }
> }
> /**
> * @ see
> spark-1.5.1/mllib/src/main/scala/org/apache/spark/ml/feature/Tokenizer.scala
> * @ see
> https://chimpler.wordpress.com/2014/06/11/classifiying-documents-using-naive-bayes-on-apache-spark-mllib/
> * @ see
> http://www.tonytruong.net/movie-rating-prediction-with-apache-spark-and-hortonworks/
> *
> * @author andrewdavidson
> *
> */
> public class Stemmer extends UnaryTransformer, List,
> Stemmer> implements Serializable{
> static Logger logger = LoggerFactory.getLogger(Stemmer.class);
> private static final long serialVersionUID = 1L;
> private static final ArrayType inputType =
> DataTypes.createArrayType(DataTypes.StringType, true);
> private final String uid = Stemmer.class.getSimpleName() + "_" +
> UUID.randomUUID().toString();
> @Override
> public String uid() {
> return uid;
> }
> /*
>override protected def validateInputType(inputType: DataType): Unit = {
> require(inputType == StringType, s"Input type must be string type but got
> $inputType.")
> }
> */
> @Override
> public void validateInputType(DataType inputTypeArg) {
> String msg = "inputType must be " + inputType.simpleString() + " but
> got " + inputTypeArg.simpleString();
> assert (inputType.equals(inputTypeArg)) : msg;
> }
>
> @Override
> public Function1, List> createTransformFunc() {
> //
> http://stackoverflow.com/questions/6545066/using-scala-from-java-passing-functions-as-parameters
> Function1, List> f = new
> AbstractFunction1, List>() {
> public List apply(List words) {
> for(String word : words) {
> logger.error("AEDWIP input word: {}", word);
> }
> return words;
> }
> };
>
> return f;
> }
> @Override
> public DataType outputDataType() {
> return DataTypes.createArrayType(DataTypes.StringType, true);
> }
> }
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28264) Revisiting Python / pandas UDF
[ https://issues.apache.org/jira/browse/SPARK-28264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16990872#comment-16990872 ] Hyukjin Kwon commented on SPARK-28264: -- [~rxin], I sent an email to dev list but leaving a comment here as well to make sure. I'll take over this since it's kind of stuck in the middle for now. I think it's worthy to make changes within 3.0. > Revisiting Python / pandas UDF > -- > > Key: SPARK-28264 > URL: https://issues.apache.org/jira/browse/SPARK-28264 > Project: Spark > Issue Type: Improvement > Components: PySpark, SQL >Affects Versions: 3.0.0 >Reporter: Reynold Xin >Assignee: Reynold Xin >Priority: Critical > > In the past two years, the pandas UDFs are perhaps the most important changes > to Spark for Python data science. However, these functionalities have evolved > organically, leading to some inconsistencies and confusions among users. This > document revisits UDF definition and naming, as a result of discussions among > Xiangrui, Li Jin, Hyukjin, and Reynold. > > See document here: > [https://docs.google.com/document/d/10Pkl-rqygGao2xQf6sddt0b-4FYK4g8qr_bXLKTL65A/edit#|https://docs.google.com/document/d/10Pkl-rqygGao2xQf6sddt0b-4FYK4g8qr_bXLKTL65A/edit] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-30167) Log4j configuration for REPL can't override the root logger properly.
Kousuke Saruta created SPARK-30167:
--
Summary: Log4j configuration for REPL can't override the root
logger properly.
Key: SPARK-30167
URL: https://issues.apache.org/jira/browse/SPARK-30167
Project: Spark
Issue Type: Bug
Components: Spark Shell
Affects Versions: 3.0.0
Reporter: Kousuke Saruta
Assignee: Kousuke Saruta
SPARK-11929 enabled REPL's log4j configuration to override root logger but
SPARK-26753 seems to have broken the feature.
You can see one example when you modifies the default log4j configuration like
as follows.
{code:java}
# Change the log level for rootCategory to DEBUG
log4j.rootCategory=DEBUG, console
...
# The log level for repl.Main remains WARN
log4j.logger.org.apache.spark.repl.Main=WARN{code}
If you launch REPL with the configuration, INFO level logs appear even though
the log level for REPL is WARN.
{code:java}
・・・
19/12/08 23:31:38 INFO Utils: Successfully started service 'sparkDriver' on
port 33083.
19/12/08 23:31:38 INFO SparkEnv: Registering MapOutputTracker
19/12/08 23:31:38 INFO SparkEnv: Registering BlockManagerMaster
19/12/08 23:31:38 INFO BlockManagerMasterEndpoint: Using
org.apache.spark.storage.DefaultTopologyMapper for getting topology information
19/12/08 23:31:38 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
19/12/08 23:31:38 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
・・・{code}
Before SPARK-26753 was applied, those INFO level logs are not shown with the
same log4j.properties.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30165) Eliminate compilation warnings
[ https://issues.apache.org/jira/browse/SPARK-30165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16990851#comment-16990851 ] Aman Omer commented on SPARK-30165: --- Hi [~maxgekk], I would like to work on some sub-tasks here. Feel free to ping me. > Eliminate compilation warnings > -- > > Key: SPARK-30165 > URL: https://issues.apache.org/jira/browse/SPARK-30165 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Maxim Gekk >Priority: Minor > Attachments: spark_warnings.txt > > > This is an umbrella ticket for sub-tasks for eliminating compilation > warnings. I dumped all warnings to the spark_warnings.txt file attached to > the ticket. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30165) Eliminate compilation warnings
[ https://issues.apache.org/jira/browse/SPARK-30165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-30165: --- Component/s: (was: Build) SQL > Eliminate compilation warnings > -- > > Key: SPARK-30165 > URL: https://issues.apache.org/jira/browse/SPARK-30165 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Maxim Gekk >Priority: Minor > Attachments: spark_warnings.txt > > > This is an umbrella ticket for sub-tasks for eliminating compilation > warnings. I dumped all warnings to the spark_warnings.txt file attached to > the ticket. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-30166) Eliminate compilation warnings in JSONOptions
Maxim Gekk created SPARK-30166:
--
Summary: Eliminate compilation warnings in JSONOptions
Key: SPARK-30166
URL: https://issues.apache.org/jira/browse/SPARK-30166
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 3.0.0
Reporter: Maxim Gekk
Scala 2.12 outputs the following warnings for JSONOptions:
{code}
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala
Warning:Warning:line (137)Java enum ALLOW_NUMERIC_LEADING_ZEROS in Java
enum Feature is deprecated: see corresponding Javadoc for more information.
factory.configure(JsonParser.Feature.ALLOW_NUMERIC_LEADING_ZEROS,
allowNumericLeadingZeros)
Warning:Warning:line (138)Java enum ALLOW_NON_NUMERIC_NUMBERS in Java enum
Feature is deprecated: see corresponding Javadoc for more information.
factory.configure(JsonParser.Feature.ALLOW_NON_NUMERIC_NUMBERS,
allowNonNumericNumbers)
Warning:Warning:line (139)Java enum ALLOW_BACKSLASH_ESCAPING_ANY_CHARACTER
in Java enum Feature is deprecated: see corresponding Javadoc for more
information.
factory.configure(JsonParser.Feature.ALLOW_BACKSLASH_ESCAPING_ANY_CHARACTER,
Warning:Warning:line (141)Java enum ALLOW_UNQUOTED_CONTROL_CHARS in Java
enum Feature is deprecated: see corresponding Javadoc for more information.
factory.configure(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS,
allowUnquotedControlChars)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30165) Eliminate compilation warnings
[ https://issues.apache.org/jira/browse/SPARK-30165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-30165: --- Description: This is an umbrella ticket for sub-tasks for eliminating compilation warnings. I dumped all warnings to the spark_warnings.txt file attached to the ticket. (was: This is an umbrella ticket for sub-tasks for eliminating compilation warnings. ) > Eliminate compilation warnings > -- > > Key: SPARK-30165 > URL: https://issues.apache.org/jira/browse/SPARK-30165 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 3.0.0 >Reporter: Maxim Gekk >Priority: Minor > Attachments: spark_warnings.txt > > > This is an umbrella ticket for sub-tasks for eliminating compilation > warnings. I dumped all warnings to the spark_warnings.txt file attached to > the ticket. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30165) Eliminate compilation warnings
[ https://issues.apache.org/jira/browse/SPARK-30165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-30165: --- Attachment: spark_warnings.txt > Eliminate compilation warnings > -- > > Key: SPARK-30165 > URL: https://issues.apache.org/jira/browse/SPARK-30165 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 3.0.0 >Reporter: Maxim Gekk >Priority: Minor > Attachments: spark_warnings.txt > > > This is an umbrella ticket for sub-tasks for eliminating compilation > warnings. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-30165) Eliminate compilation warnings
Maxim Gekk created SPARK-30165: -- Summary: Eliminate compilation warnings Key: SPARK-30165 URL: https://issues.apache.org/jira/browse/SPARK-30165 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 3.0.0 Reporter: Maxim Gekk This is an umbrella ticket for sub-tasks for eliminating compilation warnings. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
