[jira] [Resolved] (SPARK-23849) Tests for the samplingRatio option of json schema inferring
[ https://issues.apache.org/jira/browse/SPARK-23849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23849. - Resolution: Fixed Assignee: Maxim Gekk Fix Version/s: 2.4.0 > Tests for the samplingRatio option of json schema inferring > --- > > Key: SPARK-23849 > URL: https://issues.apache.org/jira/browse/SPARK-23849 > Project: Spark > Issue Type: Test > Components: SQL >Affects Versions: 2.3.0 >Reporter: Maxim Gekk >Assignee: Maxim Gekk >Priority: Minor > Fix For: 2.4.0 > > > The samplingRatio option of JSON datasource is not covered by tests. The > tests should check that only subset of input dataset is touched. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21337) SQL which has large ‘case when’ expressions may cause code generation beyond 64KB
[ https://issues.apache.org/jira/browse/SPARK-21337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fengchaoge updated SPARK-21337: --- Attachment: (was: pom.xml) > SQL which has large ‘case when’ expressions may cause code generation beyond > 64KB > - > > Key: SPARK-21337 > URL: https://issues.apache.org/jira/browse/SPARK-21337 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.1.1 > Environment: spark-2.1.1-hadoop-2.6.0-cdh-5.4.2 >Reporter: fengchaoge >Priority: Major > Fix For: 2.1.1 > > Attachments: test.JPG, test1.JPG, test2.JPG > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21337) SQL which has large ‘case when’ expressions may cause code generation beyond 64KB
[ https://issues.apache.org/jira/browse/SPARK-21337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fengchaoge updated SPARK-21337: --- Attachment: pom.xml > SQL which has large ‘case when’ expressions may cause code generation beyond > 64KB > - > > Key: SPARK-21337 > URL: https://issues.apache.org/jira/browse/SPARK-21337 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.1.1 > Environment: spark-2.1.1-hadoop-2.6.0-cdh-5.4.2 >Reporter: fengchaoge >Priority: Major > Fix For: 2.1.1 > > Attachments: pom.xml, test.JPG, test1.JPG, test2.JPG > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-23809) Active SparkSession should be set by getOrCreate
[ https://issues.apache.org/jira/browse/SPARK-23809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23809. - Resolution: Fixed Assignee: Eric Liang Fix Version/s: 2.3.1 > Active SparkSession should be set by getOrCreate > > > Key: SPARK-23809 > URL: https://issues.apache.org/jira/browse/SPARK-23809 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Eric Liang >Assignee: Eric Liang >Priority: Minor > Fix For: 2.3.1 > > > Currently, the active spark session is set inconsistently (e.g., in > createDataFrame, prior to query execution). Many places in spark also > incorrectly query active session when they should be calling > activeSession.getOrElse(defaultSession). > The semantics here can be cleaned up if we also set the active session when > the default session is set. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23809) Active SparkSession should be set by getOrCreate
[ https://issues.apache.org/jira/browse/SPARK-23809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23809: Fix Version/s: 2.4.0 > Active SparkSession should be set by getOrCreate > > > Key: SPARK-23809 > URL: https://issues.apache.org/jira/browse/SPARK-23809 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Eric Liang >Assignee: Eric Liang >Priority: Minor > Fix For: 2.3.1, 2.4.0 > > > Currently, the active spark session is set inconsistently (e.g., in > createDataFrame, prior to query execution). Many places in spark also > incorrectly query active session when they should be calling > activeSession.getOrElse(defaultSession). > The semantics here can be cleaned up if we also set the active session when > the default session is set. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-23847) Add asc_nulls_first, asc_nulls_last to PySpark
[ https://issues.apache.org/jira/browse/SPARK-23847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-23847: Assignee: Huaxin Gao (was: Huaxin Gao) > Add asc_nulls_first, asc_nulls_last to PySpark > -- > > Key: SPARK-23847 > URL: https://issues.apache.org/jira/browse/SPARK-23847 > Project: Spark > Issue Type: Sub-task > Components: PySpark, SQL >Affects Versions: 2.4.0 >Reporter: Huaxin Gao >Assignee: Huaxin Gao >Priority: Minor > Fix For: 2.4.0 > > > Column.scala and Functions.scala have asc_nulls_first, asc_nulls_last, > desc_nulls_first and desc_nulls_last. Add the corresponding python APIs in > PySpark. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-23847) Add asc_nulls_first, asc_nulls_last to PySpark
[ https://issues.apache.org/jira/browse/SPARK-23847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-23847: Assignee: Huaxin Gao > Add asc_nulls_first, asc_nulls_last to PySpark > -- > > Key: SPARK-23847 > URL: https://issues.apache.org/jira/browse/SPARK-23847 > Project: Spark > Issue Type: Sub-task > Components: PySpark, SQL >Affects Versions: 2.4.0 >Reporter: Huaxin Gao >Assignee: Huaxin Gao >Priority: Minor > Fix For: 2.4.0 > > > Column.scala and Functions.scala have asc_nulls_first, asc_nulls_last, > desc_nulls_first and desc_nulls_last. Add the corresponding python APIs in > PySpark. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-23847) Add asc_nulls_first, asc_nulls_last to PySpark
[ https://issues.apache.org/jira/browse/SPARK-23847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-23847. -- Resolution: Fixed Fix Version/s: 2.4.0 Fixed in https://github.com/apache/spark/pull/20962. > Add asc_nulls_first, asc_nulls_last to PySpark > -- > > Key: SPARK-23847 > URL: https://issues.apache.org/jira/browse/SPARK-23847 > Project: Spark > Issue Type: Sub-task > Components: PySpark, SQL >Affects Versions: 2.4.0 >Reporter: Huaxin Gao >Priority: Minor > Fix For: 2.4.0 > > > Column.scala and Functions.scala have asc_nulls_first, asc_nulls_last, > desc_nulls_first and desc_nulls_last. Add the corresponding python APIs in > PySpark. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-23866) Extend ALTER TABLE DROP PARTITION syntax to use all comparators
[ https://issues.apache.org/jira/browse/SPARK-23866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23866. - Resolution: Duplicate > Extend ALTER TABLE DROP PARTITION syntax to use all comparators > --- > > Key: SPARK-23866 > URL: https://issues.apache.org/jira/browse/SPARK-23866 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.3.0 >Reporter: Dong Jiang >Priority: Major > > Please add SQL support equivalent to drop multiple partitions by operators > other than =, basically equivalent of > https://issues.apache.org/jira/browse/HIVE-2908 > "To drop a partition from a Hive table, this works: > ALTER TABLE foo DROP PARTITION(ds = 'date') > ...but it should also work to drop all partitions prior to date. > ALTER TABLE foo DROP PARTITION(ds < 'date') > This task is to implement ALTER TABLE DROP PARTITION for all of the > comparators, < > <= >= <> = != instead of just for =." -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-14922) Alter Table Drop Partition Using Predicate-based Partition Spec
[
https://issues.apache.org/jira/browse/SPARK-14922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Li updated SPARK-14922:
Target Version/s: 2.4.0
> Alter Table Drop Partition Using Predicate-based Partition Spec
> ---
>
> Key: SPARK-14922
> URL: https://issues.apache.org/jira/browse/SPARK-14922
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.0.0, 2.1.2, 2.2.1
>Reporter: Xiao Li
>Priority: Major
>
> Below is allowed in Hive, but not allowed in Spark.
> {noformat}
> alter table ptestfilter drop partition (c='US', d<'2')
> {noformat}
> This example is copied from drop_partitions_filter.q
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23894) Flaky Test: BucketedWriteWithoutHiveSupportSuite
[
https://issues.apache.org/jira/browse/SPARK-23894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imran Rashid updated SPARK-23894:
-
Attachment: unit-tests.log
> Flaky Test: BucketedWriteWithoutHiveSupportSuite
> -
>
> Key: SPARK-23894
> URL: https://issues.apache.org/jira/browse/SPARK-23894
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.4.0
>Reporter: Imran Rashid
>Priority: Minor
> Attachments: unit-tests.log
>
>
> Flaky test observed here:
> https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88991/
> I'll attach a snippet of the unit-tests logs, for this suite and the
> preceeding one. Here's a snippet of the exception.
> {noformat}
> 08:36:34.694 Executor task launch worker for task 436 ERROR Executor:
> Exception in task 0.0 in stage 402.0 (TID 436)
> java.lang.IllegalStateException: LiveListenerBus is stopped.
> at
> org.apache.spark.scheduler.LiveListenerBus.addToQueue(LiveListenerBus.scala:97)
> at
> org.apache.spark.scheduler.LiveListenerBus.addToStatusQueue(LiveListenerBus.scala:80)
> at
> org.apache.spark.sql.internal.SharedState.(SharedState.scala:93)
> at
> org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:117)
> at
> org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:117)
> at scala.Option.getOrElse(Option.scala:121)
> at
> org.apache.spark.sql.SparkSession.sharedState$lzycompute(SparkSession.scala:117)
> at
> org.apache.spark.sql.SparkSession.sharedState(SparkSession.scala:116)
> at
> org.apache.spark.sql.internal.BaseSessionStateBuilder.build(BaseSessionStateBuilder.scala:286)
> at
> org.apache.spark.sql.test.TestSparkSession.sessionState$lzycompute(TestSQLContext.scala:42)
> at
> org.apache.spark.sql.test.TestSparkSession.sessionState(TestSQLContext.scala:41)
> at
> org.apache.spark.sql.SparkSession$$anonfun$1$$anonfun$apply$1.apply(SparkSession.scala:92)
> at
> org.apache.spark.sql.SparkSession$$anonfun$1$$anonfun$apply$1.apply(SparkSession.scala:92)
> at scala.Option.map(Option.scala:146)
> at
> org.apache.spark.sql.SparkSession$$anonfun$1.apply(SparkSession.scala:92)
> at
> org.apache.spark.sql.SparkSession$$anonfun$1.apply(SparkSession.scala:91)
> at org.apache.spark.sql.internal.SQLConf$.get(SQLConf.scala:110)
> at org.apache.spark.sql.types.DataType.sameType(DataType.scala:84)
> at
> org.apache.spark.sql.catalyst.analysis.TypeCoercion$$anonfun$1.apply(TypeCoercion.scala:105)
> at
> org.apache.spark.sql.catalyst.analysis.TypeCoercion$$anonfun$1.apply(TypeCoercion.scala:86)
> {noformat}
> I doubt this is actually because of BucketedWriteWithoutHiveSupportSuite. I
> think it has something more to do with {{SparkSession}} 's lazy evaluation of
> {{SharedState}} doing something funny with the way we setup the test spark
> context etc ... though I don't really understand it yet.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-23894) Flaky Test: BucketedWriteWithoutHiveSupportSuite
Imran Rashid created SPARK-23894:
Summary: Flaky Test: BucketedWriteWithoutHiveSupportSuite
Key: SPARK-23894
URL: https://issues.apache.org/jira/browse/SPARK-23894
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 2.4.0
Reporter: Imran Rashid
Flaky test observed here:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88991/
I'll attach a snippet of the unit-tests logs, for this suite and the preceeding
one. Here's a snippet of the exception.
{noformat}
08:36:34.694 Executor task launch worker for task 436 ERROR Executor: Exception
in task 0.0 in stage 402.0 (TID 436)
java.lang.IllegalStateException: LiveListenerBus is stopped.
at
org.apache.spark.scheduler.LiveListenerBus.addToQueue(LiveListenerBus.scala:97)
at
org.apache.spark.scheduler.LiveListenerBus.addToStatusQueue(LiveListenerBus.scala:80)
at
org.apache.spark.sql.internal.SharedState.(SharedState.scala:93)
at
org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:117)
at
org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:117)
at scala.Option.getOrElse(Option.scala:121)
at
org.apache.spark.sql.SparkSession.sharedState$lzycompute(SparkSession.scala:117)
at org.apache.spark.sql.SparkSession.sharedState(SparkSession.scala:116)
at
org.apache.spark.sql.internal.BaseSessionStateBuilder.build(BaseSessionStateBuilder.scala:286)
at
org.apache.spark.sql.test.TestSparkSession.sessionState$lzycompute(TestSQLContext.scala:42)
at
org.apache.spark.sql.test.TestSparkSession.sessionState(TestSQLContext.scala:41)
at
org.apache.spark.sql.SparkSession$$anonfun$1$$anonfun$apply$1.apply(SparkSession.scala:92)
at
org.apache.spark.sql.SparkSession$$anonfun$1$$anonfun$apply$1.apply(SparkSession.scala:92)
at scala.Option.map(Option.scala:146)
at
org.apache.spark.sql.SparkSession$$anonfun$1.apply(SparkSession.scala:92)
at
org.apache.spark.sql.SparkSession$$anonfun$1.apply(SparkSession.scala:91)
at org.apache.spark.sql.internal.SQLConf$.get(SQLConf.scala:110)
at org.apache.spark.sql.types.DataType.sameType(DataType.scala:84)
at
org.apache.spark.sql.catalyst.analysis.TypeCoercion$$anonfun$1.apply(TypeCoercion.scala:105)
at
org.apache.spark.sql.catalyst.analysis.TypeCoercion$$anonfun$1.apply(TypeCoercion.scala:86)
{noformat}
I doubt this is actually because of BucketedWriteWithoutHiveSupportSuite. I
think it has something more to do with {{SparkSession}} 's lazy evaluation of
{{SharedState}} doing something funny with the way we setup the test spark
context etc ... though I don't really understand it yet.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23871) add python api for VectorAssembler handleInvalid
[ https://issues.apache.org/jira/browse/SPARK-23871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429419#comment-16429419 ] Apache Spark commented on SPARK-23871: -- User 'huaxingao' has created a pull request for this issue: https://github.com/apache/spark/pull/21003 > add python api for VectorAssembler handleInvalid > > > Key: SPARK-23871 > URL: https://issues.apache.org/jira/browse/SPARK-23871 > Project: Spark > Issue Type: Sub-task > Components: ML, PySpark >Affects Versions: 2.3.0 >Reporter: yogesh garg >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-23871) add python api for VectorAssembler handleInvalid
[ https://issues.apache.org/jira/browse/SPARK-23871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23871: Assignee: Apache Spark > add python api for VectorAssembler handleInvalid > > > Key: SPARK-23871 > URL: https://issues.apache.org/jira/browse/SPARK-23871 > Project: Spark > Issue Type: Sub-task > Components: ML, PySpark >Affects Versions: 2.3.0 >Reporter: yogesh garg >Assignee: Apache Spark >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-23871) add python api for VectorAssembler handleInvalid
[ https://issues.apache.org/jira/browse/SPARK-23871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23871: Assignee: (was: Apache Spark) > add python api for VectorAssembler handleInvalid > > > Key: SPARK-23871 > URL: https://issues.apache.org/jira/browse/SPARK-23871 > Project: Spark > Issue Type: Sub-task > Components: ML, PySpark >Affects Versions: 2.3.0 >Reporter: yogesh garg >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23893) Possible overflow in long = int * int
[ https://issues.apache.org/jira/browse/SPARK-23893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429391#comment-16429391 ] Apache Spark commented on SPARK-23893: -- User 'kiszk' has created a pull request for this issue: https://github.com/apache/spark/pull/21002 > Possible overflow in long = int * int > - > > Key: SPARK-23893 > URL: https://issues.apache.org/jira/browse/SPARK-23893 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 2.3.0 >Reporter: Kazuaki Ishizaki >Priority: Minor > > To perform `int * int` and then to cast to `long` may cause overflow if the > MSB of the multiplication result is `1`. In other words, the result would be > negative due to sign extension. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-23893) Possible overflow in long = int * int
[ https://issues.apache.org/jira/browse/SPARK-23893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23893: Assignee: Apache Spark > Possible overflow in long = int * int > - > > Key: SPARK-23893 > URL: https://issues.apache.org/jira/browse/SPARK-23893 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 2.3.0 >Reporter: Kazuaki Ishizaki >Assignee: Apache Spark >Priority: Minor > > To perform `int * int` and then to cast to `long` may cause overflow if the > MSB of the multiplication result is `1`. In other words, the result would be > negative due to sign extension. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-23893) Possible overflow in long = int * int
[ https://issues.apache.org/jira/browse/SPARK-23893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23893: Assignee: (was: Apache Spark) > Possible overflow in long = int * int > - > > Key: SPARK-23893 > URL: https://issues.apache.org/jira/browse/SPARK-23893 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 2.3.0 >Reporter: Kazuaki Ishizaki >Priority: Minor > > To perform `int * int` and then to cast to `long` may cause overflow if the > MSB of the multiplication result is `1`. In other words, the result would be > negative due to sign extension. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-23893) Possible overflow in long = int * int
Kazuaki Ishizaki created SPARK-23893: Summary: Possible overflow in long = int * int Key: SPARK-23893 URL: https://issues.apache.org/jira/browse/SPARK-23893 Project: Spark Issue Type: Bug Components: Spark Core, SQL Affects Versions: 2.3.0 Reporter: Kazuaki Ishizaki To perform `int * int` and then to cast to `long` may cause overflow if the MSB of the multiplication result is `1`. In other words, the result would be negative due to sign extension. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-23735) Optimize the document by adding an import streaming configuration
[ https://issues.apache.org/jira/browse/SPARK-23735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-23735. --- Resolution: Won't Fix > Optimize the document by adding an import streaming configuration > -- > > Key: SPARK-23735 > URL: https://issues.apache.org/jira/browse/SPARK-23735 > Project: Spark > Issue Type: Improvement > Components: Documentation >Affects Versions: 2.2.1, 2.3.0 >Reporter: liuxianjiao >Priority: Minor > > Optimize the document by adding an import streaming > configuration，spark.streaming.concurrentJobs。This parameter is quite import > ,but is lacking in our current spark document. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-23854) Update Guava to 16.0.1
[ https://issues.apache.org/jira/browse/SPARK-23854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-23854. --- Resolution: Won't Fix > Update Guava to 16.0.1 > -- > > Key: SPARK-23854 > URL: https://issues.apache.org/jira/browse/SPARK-23854 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.3.0 >Reporter: Fokko Driesprong >Priority: Minor > > Currently Spark is still on Guava 14.0.1, and therefore I would like to bump > the version to 16.0.1. > Babysteps are important here, because we don't want to become incompatible > with other technology stacks, but 14.0.1 is getting old. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19724) create a managed table with an existed default location should throw an exception
[
https://issues.apache.org/jira/browse/SPARK-19724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429354#comment-16429354
]
Apache Spark commented on SPARK-19724:
--
User 'gengliangwang' has created a pull request for this issue:
https://github.com/apache/spark/pull/21001
> create a managed table with an existed default location should throw an
> exception
> -
>
> Key: SPARK-19724
> URL: https://issues.apache.org/jira/browse/SPARK-19724
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.2.0
>Reporter: Song Jun
>Priority: Major
>
> This JIRA is a follow up work after
> [SPARK-19583](https://issues.apache.org/jira/browse/SPARK-19583)
> As we discussed in that [PR](https://github.com/apache/spark/pull/16938)
> The following DDL for a managed table with an existed default location should
> throw an exception:
> {code}
> CREATE TABLE ... (PARTITIONED BY ...) AS SELECT ...
> CREATE TABLE ... (PARTITIONED BY ...)
> {code}
> Currently there are some situations which are not consist with above logic:
> 1. CREATE TABLE ... (PARTITIONED BY ...) succeed with an existed default
> location
> situation: for both hive/datasource(with HiveExternalCatalog/InMemoryCatalog)
> 2. CREATE TABLE ... (PARTITIONED BY ...) AS SELECT ...
> situation: hive table succeed with an existed default location
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23892) Improve coverage and fix lint error in UTF8String-related Suite
[
https://issues.apache.org/jira/browse/SPARK-23892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429253#comment-16429253
]
Apache Spark commented on SPARK-23892:
--
User 'kiszk' has created a pull request for this issue:
https://github.com/apache/spark/pull/21000
> Improve coverage and fix lint error in UTF8String-related Suite
> ---
>
> Key: SPARK-23892
> URL: https://issues.apache.org/jira/browse/SPARK-23892
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.3.0
>Reporter: Kazuaki Ishizaki
>Priority: Minor
>
> The following code in {{UTF8StringSuite}} has no sense.
> {code}
> assertTrue(s1.startsWith(s1));
> assertTrue(s1.endsWith(s1));
> {code}
> The code {{if (length <= 0) ""}} in {{UTF8StringPropertyCheckSuite}} has no
> sense
> {code}
> test("lpad, rpad") {
> def padding(origin: String, pad: String, length: Int, isLPad: Boolean):
> String = {
> if (length <= 0) return ""
> if (length <= origin.length) {
> if (length <= 0) "" else origin.substring(0, length)
> } else {
>...
> {code}
> The previous change in {{UTF8StringSuite}} broke lint-java check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-23892) Improve coverage and fix lint error in UTF8String-related Suite
[
https://issues.apache.org/jira/browse/SPARK-23892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-23892:
Assignee: (was: Apache Spark)
> Improve coverage and fix lint error in UTF8String-related Suite
> ---
>
> Key: SPARK-23892
> URL: https://issues.apache.org/jira/browse/SPARK-23892
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.3.0
>Reporter: Kazuaki Ishizaki
>Priority: Minor
>
> The following code in {{UTF8StringSuite}} has no sense.
> {code}
> assertTrue(s1.startsWith(s1));
> assertTrue(s1.endsWith(s1));
> {code}
> The code {{if (length <= 0) ""}} in {{UTF8StringPropertyCheckSuite}} has no
> sense
> {code}
> test("lpad, rpad") {
> def padding(origin: String, pad: String, length: Int, isLPad: Boolean):
> String = {
> if (length <= 0) return ""
> if (length <= origin.length) {
> if (length <= 0) "" else origin.substring(0, length)
> } else {
>...
> {code}
> The previous change in {{UTF8StringSuite}} broke lint-java check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-23892) Improve coverage and fix lint error in UTF8String-related Suite
[
https://issues.apache.org/jira/browse/SPARK-23892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-23892:
Assignee: Apache Spark
> Improve coverage and fix lint error in UTF8String-related Suite
> ---
>
> Key: SPARK-23892
> URL: https://issues.apache.org/jira/browse/SPARK-23892
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.3.0
>Reporter: Kazuaki Ishizaki
>Assignee: Apache Spark
>Priority: Minor
>
> The following code in {{UTF8StringSuite}} has no sense.
> {code}
> assertTrue(s1.startsWith(s1));
> assertTrue(s1.endsWith(s1));
> {code}
> The code {{if (length <= 0) ""}} in {{UTF8StringPropertyCheckSuite}} has no
> sense
> {code}
> test("lpad, rpad") {
> def padding(origin: String, pad: String, length: Int, isLPad: Boolean):
> String = {
> if (length <= 0) return ""
> if (length <= origin.length) {
> if (length <= 0) "" else origin.substring(0, length)
> } else {
>...
> {code}
> The previous change in {{UTF8StringSuite}} broke lint-java check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
