[jira] [Commented] (SPARK-38963) Make stage navigable from max metrics displayed in UI
[ https://issues.apache.org/jira/browse/SPARK-38963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17524761#comment-17524761 ] pavithra ramachandran commented on SPARK-38963: --- working on it > Make stage navigable from max metrics displayed in UI > - > > Key: SPARK-38963 > URL: https://issues.apache.org/jira/browse/SPARK-38963 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 3.1.1, 3.2.0, 3.3.0 >Reporter: pavithra ramachandran >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-38963) Make stage navigable from max metrics displayed in UI
pavithra ramachandran created SPARK-38963: - Summary: Make stage navigable from max metrics displayed in UI Key: SPARK-38963 URL: https://issues.apache.org/jira/browse/SPARK-38963 Project: Spark Issue Type: Improvement Components: Web UI Affects Versions: 3.2.0, 3.1.1, 3.3.0 Reporter: pavithra ramachandran -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-35877) Spark Protobuf jar has CVE issue CVE-2015-5237
[ https://issues.apache.org/jira/browse/SPARK-35877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368810#comment-17368810 ] pavithra ramachandran commented on SPARK-35877: --- i ll raise PR soon > Spark Protobuf jar has CVE issue CVE-2015-5237 > -- > > Key: SPARK-35877 > URL: https://issues.apache.org/jira/browse/SPARK-35877 > Project: Spark > Issue Type: Bug > Components: Security, Spark Core >Affects Versions: 2.4.5, 3.1.1 >Reporter: jobit mathew >Priority: Minor > > Spark Protobuf jar has CVE issue CVE-2015-5237 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-35835) Select filter query on table with struct complex type fails
[ https://issues.apache.org/jira/browse/SPARK-35835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17366386#comment-17366386 ] pavithra ramachandran commented on SPARK-35835: --- i shall raise a PR soon > Select filter query on table with struct complex type fails > --- > > Key: SPARK-35835 > URL: https://issues.apache.org/jira/browse/SPARK-35835 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.1.1 > Environment: Spark 3.1.1 >Reporter: Chetan Bhat >Priority: Minor > > [Steps]:- > From Spark beeline create a parquet or ORC table having complex type data. > Load data in the table and execute select filter query. > 0: jdbc:hive2://vm2:22550/> create table Struct_com (CUST_ID string, YEAR > int, MONTH int, AGE int, GENDER string, EDUCATED string, IS_MARRIED string, > STRUCT_INT_DOUBLE_STRING_DATE > struct,CARD_COUNT > int,DEBIT_COUNT int, CREDIT_COUNT int, DEPOSIT double, HQ_DEPOSIT double) > stored as parquet; > +-+ > | Result | > +-+ > +-+ > No rows selected (0.161 seconds) > 0: jdbc:hive2://vm2:22550/> LOAD DATA INPATH > 'hdfs://hacluster/chetan/Struct.csv' OVERWRITE INTO TABLE Struct_com; > +-+ > | Result | > +-+ > +-+ > No rows selected (1.09 seconds) > 0: jdbc:hive2://vm2:22550/> SELECT > struct_int_double_string_date.COUNTRY,struct_int_double_string_date.CHECK_DATE,struct_int_double_string_date.CHECK_DATE,struct_in > t_double_string_date.Country, SUM(struct_int_double_string_date.id) AS Sum > FROM (select * from Struct_com) SUB_QRY WHERE > struct_int_double_string_date.id > 5700 GRO UP BY > struct_int_double_string_date.COUNTRY,struct_int_double_string_date.CHECK_DATE,struct_int_double_string_date.CHECK_DATE,struct_int_double_string_date.Country > ORDER BY struct_int_double_string_date.COUNTRY > asc,struct_int_double_string_date.CHECK_DATE > asc,struct_int_double_string_date.CHECK_DATE asc, struct_int_double_stri > ng_date.Country asc; > > [Actual Issue] : - Select filter query on table with struct complex type fails > 0: jdbc:hive2://vm2:22550/> SELECT > struct_int_double_string_date.COUNTRY,struct_int_double_string_date.CHECK_DATE,struct_int_double_string_date.CHECK_DATE,struct_in > t_double_string_date.Country, SUM(struct_int_double_string_date.id) AS Sum > FROM (select * from Struct_com) SUB_QRY WHERE > struct_int_double_string_date.id > 5700 GRO UP BY > struct_int_double_string_date.COUNTRY,struct_int_double_string_date.CHECK_DATE,struct_int_double_string_date.CHECK_DATE,struct_int_double_string_date.Country > ORDER BY struct_int_double_string_date.COUNTRY > asc,struct_int_double_string_date.CHECK_DATE > asc,struct_int_double_string_date.CHECK_DATE asc, struct_int_double_stri > ng_date.Country asc; > Error: org.apache.hive.service.cli.HiveSQLException: Error running query: > org.apache.spark.sql.catalyst.errors.package$TreeNodeException: execute, tree: > Exchange rangepartitioning(COUNTRY#139896 ASC NULLS FIRST, CHECK_DATE#139897 > ASC NULLS FIRST, CHECK_DATE#139897 ASC NULLS FIRST, COUNTRY#139896 ASC NULLS > FIRST, 200 ), ENSURE_REQUIREMENTS, [id=#17161] > +- *(2) HashAggregate(keys=[_gen_alias_139928#139928, > _gen_alias_139929#139929], functions=[sum(cast(_gen_alias_139931#139931 as > bigint))], output=[COUNTRY#139896, CHECK_DATE#139897, CHECK_DATE#139898, > Country#139899, Sum#139877L]) > +- Exchange hashpartitioning(_gen_alias_139928#139928, > _gen_alias_139929#139929, 200), ENSURE_REQUIREMENTS, [id=#17157] > +- *(1) HashAggregate(keys=[_gen_alias_139928#139928, > _gen_alias_139929#139929], > functions=[partial_sum(cast(_gen_alias_139931#139931 as bigint))], output=[_g > en_alias_139928#139928, _gen_alias_139929#139929, sum#139934L]) > +- *(1) Project [STRUCT_INT_DOUBLE_STRING_DATE#139885.COUNTRY AS > _gen_alias_139928#139928, STRUCT_INT_DOUBLE_STRING_DATE#139885.CHECK_DATE AS > _gen_alias_13 9929#139929, STRUCT_INT_DOUBLE_STRING_DATE#139885.COUNTRY AS > _gen_alias_139930#139930, STRUCT_INT_DOUBLE_STRING_DATE#139885.ID AS > _gen_alias_139931#139931] > +- *(1) Filter (isnotnull(STRUCT_INT_DOUBLE_STRING_DATE#139885) AND > (STRUCT_INT_DOUBLE_STRING_DATE#139885.ID > 5700)) > +- FileScan parquet default.struct_com[STRUCT_INT_DOUBLE_STRING_DATE#139885] > Batched: false, DataFilters: [isnotnull(STRUCT_INT_DOUBLE_STRING_DATE#13 > 9885), (STRUCT_INT_DOUBLE_STRING_DATE#139885.ID > 5700)], Format: Parquet, > Location: InMemoryFileIndex[hdfs://hacluster/user/hive/warehouse/struct_com], > PartitionFi lters: [], PushedFilters: > [IsNotNull(STRUCT_INT_DOUBLE_STRING_DATE), > GreaterThan(STRUCT_INT_DOUBLE_STRING_DATE.ID,5700)], ReadSchema: > struct G_DATE:struct> > at >
[jira] [Commented] (SPARK-32351) Partially pushed partition filters are not explained
[
https://issues.apache.org/jira/browse/SPARK-32351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17162477#comment-17162477
]
pavithra ramachandran commented on SPARK-32351:
---
i would like to check this
> Partially pushed partition filters are not explained
>
>
> Key: SPARK-32351
> URL: https://issues.apache.org/jira/browse/SPARK-32351
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Yuming Wang
>Priority: Major
>
> How to reproduce this issue:
> {code:scala}
> spark.sql(
> s"""
> |CREATE TABLE t(i INT, p STRING)
> |USING parquet
> |PARTITIONED BY (p)""".stripMargin)
> spark.range(0, 1000).selectExpr("id as col").createOrReplaceTempView("temp")
> for (part <- Seq(1, 2, 3, 4)) {
> sql(s"""
> |INSERT OVERWRITE TABLE t PARTITION (p='$part')
> |SELECT col FROM temp""".stripMargin)
> }
> spark.sql("SELECT * FROM t WHERE WHERE (p = '1' AND i = 1) OR (p = '2' and i
> = 2)").explain
> {code}
> We have pushed down {{p = '1' or p = '2'}} since SPARK-28169, but this pushed
> down filter not in explain
> {noformat}
> == Physical Plan ==
> *(1) Filter (((p#21 = 1) AND (i#20 = 1)) OR ((p#21 = 2) AND (i#20 = 2)))
> +- *(1) ColumnarToRow
>+- FileScan parquet default.t[i#20,p#21] Batched: true, DataFilters: [],
> Format: Parquet, Location:
> InMemoryFileIndex[file:/Users/yumwang/spark/SPARK-32289/sql/core/spark-warehouse/org.apache.spark...,
> PartitionFilters: [], PushedFilters: [], ReadSchema: struct
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32325) JSON predicate pushdown for nested fields
[ https://issues.apache.org/jira/browse/SPARK-32325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158878#comment-17158878 ] pavithra ramachandran commented on SPARK-32325: --- i would like to work on this > JSON predicate pushdown for nested fields > - > > Key: SPARK-32325 > URL: https://issues.apache.org/jira/browse/SPARK-32325 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: Maxim Gekk >Priority: Major > > SPARK-30648 should support filters pushdown to JSON datasource but it > supports only filters that refer to top-level fields. The ticket aims to > support nested fields as well. See the needed changes: > https://github.com/apache/spark/pull/27366#discussion_r443340603 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32328) Avro predicate pushdown for nested fields
[ https://issues.apache.org/jira/browse/SPARK-32328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158877#comment-17158877 ] pavithra ramachandran commented on SPARK-32328: --- i would like to work on this > Avro predicate pushdown for nested fields > - > > Key: SPARK-32328 > URL: https://issues.apache.org/jira/browse/SPARK-32328 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: jobit mathew >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32322) Pyspark not launching in Spark IPV6 environment
[ https://issues.apache.org/jira/browse/SPARK-32322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158165#comment-17158165 ] pavithra ramachandran commented on SPARK-32322: --- i would like to check this > Pyspark not launching in Spark IPV6 environment > --- > > Key: SPARK-32322 > URL: https://issues.apache.org/jira/browse/SPARK-32322 > Project: Spark > Issue Type: Bug > Components: PySpark >Affects Versions: 3.1.0 >Reporter: jobit mathew >Priority: Minor > > pyspark is not launching in Spark IPV6 environment. > Initial analysis looks like python is not supporting IPV6. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Issue Comment Deleted] (SPARK-31887) Date casting to string is giving wrong value
[
https://issues.apache.org/jira/browse/SPARK-31887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
pavithra ramachandran updated SPARK-31887:
--
Comment: was deleted
(was: i would like to check this.)
> Date casting to string is giving wrong value
>
>
> Key: SPARK-31887
> URL: https://issues.apache.org/jira/browse/SPARK-31887
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.4.5
> Environment: The spark is running on cluster mode with Mesos.
>
> Mesos agents are dockerised running on Ubuntu 18.
>
> Timezone setting of docker instance: UTC
> Timezone of server hosting docker: America/New_York
> Timezone of driver machine: America/New_York
>Reporter: Amit Gupta
>Priority: Major
>
> The code converts the string to date and then write it in csv.
> {code:java}
> val x = Seq(("2020-02-19", "2020-02-19 05:11:00")).toDF("a",
> "b").select('a.cast("date"), 'b.cast("timestamp"))
> x.show()
> +--+---+
> | a| b|
> +--+---+
> |2020-02-19|2020-02-19 05:11:00|
> +--+---+
> x.write.mode("overwrite").option("header", true).csv("/tmp/test1.csv")
> {code}
>
> The date written in CSV file is different:
> {code:java}
> > snakebite cat "/tmp/test1.csv/*.csv"
> a,b
> 2020-02-18,2020-02-19T05:11:00.000Z{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32103) Spark support IPV6 in yarn mode
[ https://issues.apache.org/jira/browse/SPARK-32103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146073#comment-17146073 ] pavithra ramachandran commented on SPARK-32103: --- i shall raise a pr soon > Spark support IPV6 in yarn mode > --- > > Key: SPARK-32103 > URL: https://issues.apache.org/jira/browse/SPARK-32103 > Project: Spark > Issue Type: Bug > Components: YARN >Affects Versions: 3.1.0 >Reporter: jobit mathew >Priority: Minor > > Spark support IPV6 in yarn mode > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31908) Spark UI shows wrong driver memory configurations if driver memory is provided at run time
[ https://issues.apache.org/jira/browse/SPARK-31908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126419#comment-17126419 ] pavithra ramachandran commented on SPARK-31908: --- i would like to work on this > Spark UI shows wrong driver memory configurations if driver memory is > provided at run time > -- > > Key: SPARK-31908 > URL: https://issues.apache.org/jira/browse/SPARK-31908 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.4.3 >Reporter: Rahul Kumar >Priority: Major > > I discovered that in cluster mode , when driver memory is provided via > spark.driver.memory configuration at run time after creating spark session, > spark doesn't pick this configurations at run time as application master is > already launched by that time and picks the default spark driver memory > configuration(1GB). > However, on spark UI page, in environment tab, it still shows driver memory > as the value passed via configurations at run time, which makes identifying > and debugging this scenario more difficult. Driver memory should be shown as > the value which spark is actually using in job. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31887) Date casting to string is giving wrong value
[
https://issues.apache.org/jira/browse/SPARK-31887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123835#comment-17123835
]
pavithra ramachandran commented on SPARK-31887:
---
i would like to check this.
> Date casting to string is giving wrong value
>
>
> Key: SPARK-31887
> URL: https://issues.apache.org/jira/browse/SPARK-31887
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.4.5
> Environment: The spark is running on cluster mode with Mesos.
>
> Mesos agents are dockerised running on Ubuntu 18.
>
> Timezone setting of docker instance: UTC
> Timezone of server hosting docker: America/New_York
> Timezone of driver machine: America/New_York
>Reporter: Amit Gupta
>Priority: Major
>
> The code converts the string to date and then write it in csv.
> {code:java}
> val x = Seq(("2020-02-19", "2020-02-19 05:11:00")).toDF("a",
> "b").select('a.cast("date"), 'b.cast("timestamp"))
> x.show()
> +--+---+
> | a| b|
> +--+---+
> |2020-02-19|2020-02-19 05:11:00|
> +--+---+
> x.write.mode("overwrite").option("header", true).csv("/tmp/test1.csv")
> {code}
>
> The date written in CSV file is different:
> {code:java}
> > snakebite cat "/tmp/test1.csv/*.csv"
> a,b
> 2020-02-18,2020-02-19T05:11:00.000Z{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31634) "show tables like" support for SQL wildcard characters (% and _)
[ https://issues.apache.org/jira/browse/SPARK-31634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104223#comment-17104223 ] pavithra ramachandran commented on SPARK-31634: --- [~yumwang] i see that show tables uses catalogue and there is an open Jira in hive side. Once that gets fixed, it will work in spark, Or do u want us to handle separately handle in spark. > "show tables like" support for SQL wildcard characters (% and _) > > > Key: SPARK-31634 > URL: https://issues.apache.org/jira/browse/SPARK-31634 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.0 >Reporter: Yuming Wang >Priority: Major > > https://docs.snowflake.com/en/sql-reference/sql/show-tables.html > https://clickhouse.tech/docs/en/sql-reference/statements/show/ > https://www.mysqltutorial.org/mysql-show-tables/ > https://issues.apache.org/jira/browse/HIVE-23359 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30168) Eliminate warnings in Parquet datasource
[
https://issues.apache.org/jira/browse/SPARK-30168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103181#comment-17103181
]
pavithra ramachandran commented on SPARK-30168:
---
[~maxgekk] I would like to work this, i shall raise the PR soon
> Eliminate warnings in Parquet datasource
>
>
> Key: SPARK-30168
> URL: https://issues.apache.org/jira/browse/SPARK-30168
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.1.0
>Reporter: Maxim Gekk
>Priority: Minor
>
> #
> sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetPartitionReaderFactory.scala
> {code}
> Warning:Warning:line (120)class ParquetInputSplit in package hadoop is
> deprecated: see corresponding Javadoc for more information.
> Option[TimeZone]) => RecordReader[Void, T]): RecordReader[Void, T]
> = {
> Warning:Warning:line (125)class ParquetInputSplit in package hadoop is
> deprecated: see corresponding Javadoc for more information.
> new org.apache.parquet.hadoop.ParquetInputSplit(
> Warning:Warning:line (134)method readFooter in class ParquetFileReader is
> deprecated: see corresponding Javadoc for more information.
> ParquetFileReader.readFooter(conf, filePath,
> SKIP_ROW_GROUPS).getFileMetaData
> Warning:Warning:line (183)class ParquetInputSplit in package hadoop is
> deprecated: see corresponding Javadoc for more information.
> split: ParquetInputSplit,
> Warning:Warning:line (212)class ParquetInputSplit in package hadoop is
> deprecated: see corresponding Javadoc for more information.
> split: ParquetInputSplit,
> {code}
> #
> sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
> {code}
> Warning:Warning:line (55)java: org.apache.parquet.hadoop.ParquetInputSplit in
> org.apache.parquet.hadoop has been deprecated
> Warning:Warning:line (95)java:
> org.apache.parquet.hadoop.ParquetInputSplit in org.apache.parquet.hadoop has
> been deprecated
> Warning:Warning:line (95)java:
> org.apache.parquet.hadoop.ParquetInputSplit in org.apache.parquet.hadoop has
> been deprecated
> Warning:Warning:line (97)java: getRowGroupOffsets() in
> org.apache.parquet.hadoop.ParquetInputSplit has been deprecated
> Warning:Warning:line (105)java:
> readFooter(org.apache.hadoop.conf.Configuration,org.apache.hadoop.fs.Path,org.apache.parquet.format.converter.ParquetMetadataConverter.MetadataFilter)
> in org.apache.parquet.hadoop.ParquetFileReader has been deprecated
> Warning:Warning:line (108)java:
> filterRowGroups(org.apache.parquet.filter2.compat.FilterCompat.Filter,java.util.List,org.apache.parquet.schema.MessageType)
> in org.apache.parquet.filter2.compat.RowGroupFilter has been deprecated
> Warning:Warning:line (111)java:
> readFooter(org.apache.hadoop.conf.Configuration,org.apache.hadoop.fs.Path,org.apache.parquet.format.converter.ParquetMetadataConverter.MetadataFilter)
> in org.apache.parquet.hadoop.ParquetFileReader has been deprecated
> Warning:Warning:line (147)java:
> ParquetFileReader(org.apache.hadoop.conf.Configuration,org.apache.parquet.hadoop.metadata.FileMetaData,org.apache.hadoop.fs.Path,java.util.List,java.util.List)
> in org.apache.parquet.hadoop.ParquetFileReader has been deprecated
> Warning:Warning:line (203)java:
> readFooter(org.apache.hadoop.conf.Configuration,org.apache.hadoop.fs.Path,org.apache.parquet.format.converter.ParquetMetadataConverter.MetadataFilter)
> in org.apache.parquet.hadoop.ParquetFileReader has been deprecated
> Warning:Warning:line (226)java:
> ParquetFileReader(org.apache.hadoop.conf.Configuration,org.apache.parquet.hadoop.metadata.FileMetaData,org.apache.hadoop.fs.Path,java.util.List,java.util.List)
> in org.apache.parquet.hadoop.ParquetFileReader has been deprecated
> {code}
> #
> sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetCompatibilityTest.scala
> #
> sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetInteroperabilitySuite.scala
> #
> sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTest.scala
> #
> sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31634) "show tables like" support for SQL wildcard characters (% and _)
[ https://issues.apache.org/jira/browse/SPARK-31634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099984#comment-17099984 ] pavithra ramachandran commented on SPARK-31634: --- I would like to work on this > "show tables like" support for SQL wildcard characters (% and _) > > > Key: SPARK-31634 > URL: https://issues.apache.org/jira/browse/SPARK-31634 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.0 >Reporter: Yuming Wang >Priority: Major > > https://docs.snowflake.com/en/sql-reference/sql/show-tables.html > https://clickhouse.tech/docs/en/sql-reference/statements/show/ > https://www.mysqltutorial.org/mysql-show-tables/ > https://issues.apache.org/jira/browse/HIVE-23359 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-31634) "show tables like" support for SQL wildcard characters (% and _)
[ https://issues.apache.org/jira/browse/SPARK-31634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099984#comment-17099984 ] pavithra ramachandran edited comment on SPARK-31634 at 5/5/20, 3:06 PM: I would like to work on this. [~yumwang] can I ? was (Author: pavithraramachandran): I would like to work on this > "show tables like" support for SQL wildcard characters (% and _) > > > Key: SPARK-31634 > URL: https://issues.apache.org/jira/browse/SPARK-31634 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.0 >Reporter: Yuming Wang >Priority: Major > > https://docs.snowflake.com/en/sql-reference/sql/show-tables.html > https://clickhouse.tech/docs/en/sql-reference/statements/show/ > https://www.mysqltutorial.org/mysql-show-tables/ > https://issues.apache.org/jira/browse/HIVE-23359 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30687) When reading from a file with pre-defined schema and encountering a single value that is not the same type as that of its column , Spark nullifies the entire row
[
https://issues.apache.org/jira/browse/SPARK-30687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029720#comment-17029720
]
pavithra ramachandran commented on SPARK-30687:
---
yes. Issue is present 2.4.x also.
> When reading from a file with pre-defined schema and encountering a single
> value that is not the same type as that of its column , Spark nullifies the
> entire row
> -
>
> Key: SPARK-30687
> URL: https://issues.apache.org/jira/browse/SPARK-30687
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.3.0
>Reporter: Bao Nguyen
>Priority: Major
>
> When reading from a file with pre-defined schema and encountering a single
> value that is not the same type as that of its column , Spark nullifies the
> entire row instead of setting the value at that cell to be null.
>
> {code:java}
> case class TestModel(
> num: Double, test: String, mac: String, value: Double
> )
> val schema =
> ScalaReflection.schemaFor[TestModel].dataType.asInstanceOf[StructType]
> //here's the content of the file test.data
> //1~test~mac1~2
> //1.0~testdatarow2~mac2~non-numeric
> //2~test1~mac1~3
> val ds = spark
> .read
> .schema(schema)
> .option("delimiter", "~")
> .csv("/test-data/test.data")
> ds.show();
> //the content of data frame. second row is all null.
> // ++-++-+
> // | num| test| mac|value|
> // ++-++-+
> // | 1.0| test|mac1| 2.0|
> // |null| null|null| null|
> // | 2.0|test1|mac1| 3.0|
> // ++-++-+
> //should be
> // ++--++-+
> // | num| test | mac|value|
> // ++--++-+
> // | 1.0| test |mac1| 2.0 |
> // |1.0 |testdatarow2 |mac2| null|
> // | 2.0|test1 |mac1| 3.0 |
> // ++--++-+{code}
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30687) When reading from a file with pre-defined schema and encountering a single value that is not the same type as that of its column , Spark nullifies the entire row
[
https://issues.apache.org/jira/browse/SPARK-30687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027333#comment-17027333
]
pavithra ramachandran commented on SPARK-30687:
---
l would like to work on this issue.
> When reading from a file with pre-defined schema and encountering a single
> value that is not the same type as that of its column , Spark nullifies the
> entire row
> -
>
> Key: SPARK-30687
> URL: https://issues.apache.org/jira/browse/SPARK-30687
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.3.0
>Reporter: Bao Nguyen
>Priority: Major
>
> When reading from a file with pre-defined schema and encountering a single
> value that is not the same type as that of its column , Spark nullifies the
> entire row instead of setting the value at that cell to be null.
>
> {code:java}
> case class TestModel(
> num: Double, test: String, mac: String, value: Double
> )
> val schema =
> ScalaReflection.schemaFor[TestModel].dataType.asInstanceOf[StructType]
> //here's the content of the file test.data
> //1~test~mac1~2
> //1.0~testdatarow2~mac2~non-numeric
> //2~test1~mac1~3
> val ds = spark
> .read
> .schema(schema)
> .option("delimiter", "~")
> .csv("/test-data/test.data")
> ds.show();
> //the content of data frame. second row is all null.
> // ++-++-+
> // | num| test| mac|value|
> // ++-++-+
> // | 1.0| test|mac1| 2.0|
> // |null| null|null| null|
> // | 2.0|test1|mac1| 3.0|
> // ++-++-+
> //should be
> // ++--++-+
> // | num| test | mac|value|
> // ++--++-+
> // | 1.0| test |mac1| 2.0 |
> // |1.0 |testdatarow2 |mac2| null|
> // | 2.0|test1 |mac1| 3.0 |
> // ++--++-+{code}
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30483) Job History does not show pool properties table
[ https://issues.apache.org/jira/browse/SPARK-30483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016873#comment-17016873 ] pavithra ramachandran commented on SPARK-30483: --- Issue is resolved in master and 2.4 [https://github.com/apache/spark/commit/6d90298438e627187088a5d8c53d470646d051f4] > Job History does not show pool properties table > --- > > Key: SPARK-30483 > URL: https://issues.apache.org/jira/browse/SPARK-30483 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 3.0.0 >Reporter: ABHISHEK KUMAR GUPTA >Priority: Major > > Stage will show the Pool Name column but when user clicks the hyper link Name> it will not redirect to Pool Properties Table. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30484) Job History Storage Tab does not display RDD Table
[
https://issues.apache.org/jira/browse/SPARK-30484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17012656#comment-17012656
]
pavithra ramachandran commented on SPARK-30484:
---
i shall work on this
> Job History Storage Tab does not display RDD Table
> --
>
> Key: SPARK-30484
> URL: https://issues.apache.org/jira/browse/SPARK-30484
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 3.0.0
>Reporter: ABHISHEK KUMAR GUPTA
>Priority: Major
>
> scala> import org.apache.spark.storage.StorageLevel._
> import org.apache.spark.storage.StorageLevel._
> scala> val rdd = sc.range(0, 100, 1, 5).setName("rdd")
> rdd: org.apache.spark.rdd.RDD[Long] = rdd MapPartitionsRDD[1] at range at
> :27
> scala> rdd.persist(MEMORY_ONLY_SER)
> res0: rdd.type = rdd MapPartitionsRDD[1] at range at :27
> scala> rdd.count
> res1: Long = 100
>
> scala> val df = Seq((1, "andy"), (2, "bob"), (2, "andy")).toDF("count",
> "name")
> df: org.apache.spark.sql.DataFrame = [count: int, name: string]
> scala> df.persist(DISK_ONLY)
> res2: df.type = [count: int, name: string]
> scala> df.count
> res3: Long = 3
> Open Storage Tab under Incomplete Jobs in Job History Page
> UI will not display the RDD Table.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30483) Job History does not show pool properties table
[ https://issues.apache.org/jira/browse/SPARK-30483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17012654#comment-17012654 ] pavithra ramachandran commented on SPARK-30483: --- i shall work on this > Job History does not show pool properties table > --- > > Key: SPARK-30483 > URL: https://issues.apache.org/jira/browse/SPARK-30483 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 3.0.0 >Reporter: ABHISHEK KUMAR GUPTA >Priority: Major > > Stage will show the Pool Name column but when user clicks the hyper link Name> it will not redirect to Pool Properties Table. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27148) Support CURRENT_TIME and LOCALTIME when ANSI mode enabled
[
https://issues.apache.org/jira/browse/SPARK-27148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17012620#comment-17012620
]
pavithra ramachandran commented on SPARK-27148:
---
[~maropu] I would like to work on this..
> Support CURRENT_TIME and LOCALTIME when ANSI mode enabled
> -
>
> Key: SPARK-27148
> URL: https://issues.apache.org/jira/browse/SPARK-27148
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Takeshi Yamamuro
>Priority: Major
>
> CURRENT_TIME and LOCALTIME should be supported in the ANSI standard;
> {code:java}
> postgres=# select CURRENT_TIME;
> timetz
>
> 16:45:43.398109+09
> (1 row)
> postgres=# select LOCALTIME;
> time
>
> 16:45:48.60969
> (1 row){code}
> Before this, we need to support TIME types (java.sql.Time).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30454) Null Dereference in HiveSQLException
[ https://issues.apache.org/jira/browse/SPARK-30454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010412#comment-17010412 ] pavithra ramachandran commented on SPARK-30454: --- I shall raise the PR > Null Dereference in HiveSQLException > > > Key: SPARK-30454 > URL: https://issues.apache.org/jira/browse/SPARK-30454 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.4, 2.4.4, 3.0.0 >Reporter: pavithra ramachandran >Priority: Major > > Null Pointer DeReferencing found in spark HiveSQLException code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-30454) Null Dereference in HiveSQLException
pavithra ramachandran created SPARK-30454: - Summary: Null Dereference in HiveSQLException Key: SPARK-30454 URL: https://issues.apache.org/jira/browse/SPARK-30454 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.4.4, 2.3.4, 3.0.0 Reporter: pavithra ramachandran Null Pointer DeReferencing found in spark HiveSQLException code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28825) Document EXPLAIN Statement in SQL Reference.
[ https://issues.apache.org/jira/browse/SPARK-28825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17000593#comment-17000593 ] pavithra ramachandran commented on SPARK-28825: --- [~LI,Xiao] and [~dkbiswal] i have the PR ready. Can I handle this? > Document EXPLAIN Statement in SQL Reference. > > > Key: SPARK-28825 > URL: https://issues.apache.org/jira/browse/SPARK-28825 > Project: Spark > Issue Type: Sub-task > Components: Documentation, SQL >Affects Versions: 2.4.3 >Reporter: jobit mathew >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29505) desc extended is case sensitive
[
https://issues.apache.org/jira/browse/SPARK-29505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997090#comment-16997090
]
pavithra ramachandran commented on SPARK-29505:
---
i will work on this
> desc extended is case sensitive
> --
>
> Key: SPARK-29505
> URL: https://issues.apache.org/jira/browse/SPARK-29505
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: ABHISHEK KUMAR GUPTA
>Priority: Major
>
> {code}
> create table customer(id int, name String, *CName String*, address String,
> city String, pin int, country String);
> insert into customer values(1,'Alfred','Maria','Obere Str
> 57','Berlin',12209,'Germany');
> insert into customer values(2,'Ana','trujilo','Adva de la','Maxico
> D.F.',05021,'Maxico');
> insert into customer values(3,'Antonio','Antonio Moreno','Mataderos
> 2312','Maxico D.F.',05023,'Maxico');
> analyze table customer compute statistics for columns cname; – *Success(
> Though cname is not as CName)*
> desc extended customer cname; – Failed
> jdbc:hive2://10.18.19.208:23040/default> desc extended customer *cname;*
> +-+-+
> | info_name | info_value |
> +-+-+
> | col_name | cname |
> | data_type | string |
> | comment | NULL |
> | min | NULL |
> | max | NULL |
> | num_nulls | NULL |
> | distinct_count | NULL |
> | avg_col_len | NULL |
> | max_col_len | NULL |
> | histogram | NULL |
> +-+--
> {code}
>
> But
> {code}
> desc extended customer CName; – SUCCESS
> 0: jdbc:hive2://10.18.19.208:23040/default> desc extended customer *CName;*
> +-+-+
> | info_name | info_value |
> +-+-+
> | col_name | CName |
> | data_type | string |
> | comment | NULL |
> | min | NULL |
> | max | NULL |
> | num_nulls | 0 |
> | distinct_count | 3 |
> | avg_col_len | 9 |
> | max_col_len | 14 |
> | histogram | NULL |
> +-+-+
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28825) Document EXPLAIN Statement in SQL Reference.
[ https://issues.apache.org/jira/browse/SPARK-28825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16995531#comment-16995531 ] pavithra ramachandran commented on SPARK-28825: --- [~dkbiswal] are you working on this? If not I would like to work on this. > Document EXPLAIN Statement in SQL Reference. > > > Key: SPARK-28825 > URL: https://issues.apache.org/jira/browse/SPARK-28825 > Project: Spark > Issue Type: Sub-task > Components: Documentation, SQL >Affects Versions: 2.4.3 >Reporter: jobit mathew >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29455) Improve tooltip information for Stages Tab
[ https://issues.apache.org/jira/browse/SPARK-29455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16994497#comment-16994497 ] pavithra ramachandran commented on SPARK-29455: --- [~srowen] thank you.. i see sharan has already submitted the patch . > Improve tooltip information for Stages Tab > -- > > Key: SPARK-29455 > URL: https://issues.apache.org/jira/browse/SPARK-29455 > Project: Spark > Issue Type: Sub-task > Components: Web UI >Affects Versions: 3.0.0 >Reporter: ABHISHEK KUMAR GUPTA >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30233) Spark WebUI task table indentation issue
[ https://issues.apache.org/jira/browse/SPARK-30233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16994446#comment-16994446 ] pavithra ramachandran commented on SPARK-30233: --- its resolved in master using SPARK-28430 [https://github.com/apache/spark/pull/25183] > Spark WebUI task table indentation issue > - > > Key: SPARK-30233 > URL: https://issues.apache.org/jira/browse/SPARK-30233 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.3.4 >Reporter: jobit mathew >Priority: Minor > Attachments: sparkopensourceissue.PNG > > > !sparkopensourceissue.PNG! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Issue Comment Deleted] (SPARK-30233) Spark WebUI task table indentation issue
[ https://issues.apache.org/jira/browse/SPARK-30233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pavithra ramachandran updated SPARK-30233: -- Comment: was deleted (was: i shall raise PR soon) > Spark WebUI task table indentation issue > - > > Key: SPARK-30233 > URL: https://issues.apache.org/jira/browse/SPARK-30233 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.3.4 >Reporter: jobit mathew >Priority: Minor > Attachments: sparkopensourceissue.PNG > > > !sparkopensourceissue.PNG! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30233) Spark WebUI task table indentation issue
[ https://issues.apache.org/jira/browse/SPARK-30233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16994426#comment-16994426 ] pavithra ramachandran commented on SPARK-30233: --- i shall raise PR soon > Spark WebUI task table indentation issue > - > > Key: SPARK-30233 > URL: https://issues.apache.org/jira/browse/SPARK-30233 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 3.0.0 >Reporter: jobit mathew >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30171) Eliminate warnings: part2
[
https://issues.apache.org/jira/browse/SPARK-30171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991164#comment-16991164
]
pavithra ramachandran commented on SPARK-30171:
---
i will work on this
> Eliminate warnings: part2
> -
>
> Key: SPARK-30171
> URL: https://issues.apache.org/jira/browse/SPARK-30171
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: ABHISHEK KUMAR GUPTA
>Priority: Minor
>
> AvroFunctionsSuite.scala
> Warning:Warning:line (41)method to_avro in package avro is deprecated (since
> 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro' instead.
> val avroDF = df.select(to_avro('id).as("a"), to_avro('str).as("b"))
> Warning:Warning:line (41)method to_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro'
> instead.
> val avroDF = df.select(to_avro('id).as("a"), to_avro('str).as("b"))
> Warning:Warning:line (54)method from_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
> instead.
> checkAnswer(avroDF.select(from_avro('a, avroTypeLong), from_avro('b,
> avroTypeStr)), df)
> Warning:Warning:line (54)method from_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
> instead.
> checkAnswer(avroDF.select(from_avro('a, avroTypeLong), from_avro('b,
> avroTypeStr)), df)
> Warning:Warning:line (59)method to_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro'
> instead.
> val avroStructDF = df.select(to_avro('struct).as("avro"))
> Warning:Warning:line (70)method from_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
> instead.
> checkAnswer(avroStructDF.select(from_avro('avro, avroTypeStruct)), df)
> Warning:Warning:line (76)method to_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro'
> instead.
> val avroStructDF = df.select(to_avro('struct).as("avro"))
> Warning:Warning:line (118)method to_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.to_avro'
> instead.
> val readBackOne = dfOne.select(to_avro($"array").as("avro"))
> Warning:Warning:line (119)method from_avro in package avro is deprecated
> (since 3.0.0): Please use 'org.apache.spark.sql.avro.functions.from_avro'
> instead.
> .select(from_avro($"avro", avroTypeArrStruct).as("array"))
> AvroPartitionReaderFactory.scala
> Warning:Warning:line (64)value ignoreExtension in class AvroOptions is
> deprecated (since 3.0): Use the general data source option pathGlobFilter for
> filtering file names
> if (parsedOptions.ignoreExtension ||
> partitionedFile.filePath.endsWith(".avro")) {
> AvroFileFormat.scala
> Warning:Warning:line (98)value ignoreExtension in class AvroOptions is
> deprecated (since 3.0): Use the general data source option pathGlobFilter for
> filtering file names
> if (parsedOptions.ignoreExtension || file.filePath.endsWith(".avro")) {
> AvroUtils.scala
> Warning:Warning:line (55)value ignoreExtension in class AvroOptions is
> deprecated (since 3.0): Use the general data source option pathGlobFilter for
> filtering file names
> inferAvroSchemaFromFiles(files, conf, parsedOptions.ignoreExtension,
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29841) PostgreSQL dialect: cast to date
[ https://issues.apache.org/jira/browse/SPARK-29841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971470#comment-16971470 ] pavithra ramachandran commented on SPARK-29841: --- i will check > PostgreSQL dialect: cast to date > > > Key: SPARK-29841 > URL: https://issues.apache.org/jira/browse/SPARK-29841 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.0.0 >Reporter: jobit mathew >Priority: Minor > > Make SparkSQL's cast to date behavior be consistent with PostgreSQL when > spark.sql.dialect is configured as PostgreSQL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29784) Built in function trim is not compatible in 3.0 with previous version
[
https://issues.apache.org/jira/browse/SPARK-29784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968949#comment-16968949
]
pavithra ramachandran commented on SPARK-29784:
---
i shall work on this
> Built in function trim is not compatible in 3.0 with previous version
> -
>
> Key: SPARK-29784
> URL: https://issues.apache.org/jira/browse/SPARK-29784
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: ABHISHEK KUMAR GUPTA
>Priority: Major
>
> SELECT trim('SL', 'SSparkSQLS'); returns empty in Spark 3.0 where as in 2.4
> and 2.3.2 is returning after leading and trailing character removed.
> Spark 3.0 – Not correct
> jdbc:hive2://10.18.19.208:23040/default> SELECT trim('SL', 'SSparkSQLS');
> +---+
> | trim(SL, SSparkSQLS) |
> +---+
> | |
> +---
> Spark 2.4 – Correct
> jdbc:hive2://10.18.18.214:23040/default> SELECT trim('SL', 'SSparkSQLS');
> +---+--+
> | trim(SSparkSQLS, SL) |
> +---+--+
> | parkSQ |
> +---+--+
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-28794) Document CREATE TABLE in SQL Reference.
[ https://issues.apache.org/jira/browse/SPARK-28794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966433#comment-16966433 ] pavithra ramachandran edited comment on SPARK-28794 at 11/7/19 3:08 AM: i shall raise PR by weekend. was (Author: pavithraramachandran): i shall raise PR by tomorrow. > Document CREATE TABLE in SQL Reference. > --- > > Key: SPARK-28794 > URL: https://issues.apache.org/jira/browse/SPARK-28794 > Project: Spark > Issue Type: Sub-task > Components: Documentation, SQL >Affects Versions: 2.4.3 >Reporter: Dilip Biswal >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28794) Document CREATE TABLE in SQL Reference.
[ https://issues.apache.org/jira/browse/SPARK-28794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966433#comment-16966433 ] pavithra ramachandran commented on SPARK-28794: --- i shall raise PR by tomorrow. > Document CREATE TABLE in SQL Reference. > --- > > Key: SPARK-28794 > URL: https://issues.apache.org/jira/browse/SPARK-28794 > Project: Spark > Issue Type: Sub-task > Components: Documentation, SQL >Affects Versions: 2.4.3 >Reporter: Dilip Biswal >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29601) JDBC ODBC Tab Statement column should be provided ellipsis for bigger SQL statement
[ https://issues.apache.org/jira/browse/SPARK-29601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959479#comment-16959479 ] pavithra ramachandran commented on SPARK-29601: --- i shall work on this > JDBC ODBC Tab Statement column should be provided ellipsis for bigger SQL > statement > --- > > Key: SPARK-29601 > URL: https://issues.apache.org/jira/browse/SPARK-29601 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.0.0 >Reporter: ABHISHEK KUMAR GUPTA >Priority: Major > > Statement column in JDBC/ODBC gives whole SQL statement and page size > increases. > Suppose user submit and TPCDS Queries, then Page it display whole Query under > statement and User Experience is not good. > Expected: > It should display the ... Ellipsis and on clicking the stmt. it should Expand > display the whole SQL Statement. > e.g. > SELECT * FROM (SELECT count(*) h8_30_to_9 FROM store_sales, > household_demographics, time_dim, store WHERE ss_sold_time_sk = > time_dim.t_time_sk AND ss_hdemo_sk = household_demographics.hd_demo_sk AND > ss_store_sk = s_store_sk AND time_dim.t_hour = 8 AND time_dim.t_minute >= 30 > AND ((household_demographics.hd_dep_count = 4 AND > household_demographics.hd_vehicle_count <= 4 + 2) OR > (household_demographics.hd_dep_count = 2 AND > household_demographics.hd_vehicle_count <= 2 + 2) OR > (household_demographics.hd_dep_count = 0 AND > household_demographics.hd_vehicle_count <= 0 + 2)) AND store.s_store_name = > 'ese') s1, (SELECT count(*) h9_to_9_30 FROM store_sales, > household_demographics, time_dim, store WHERE ss_sold_time_sk = > time_dim.t_time_sk AND ss_hdemo_sk = household_demographics.hd_demo_sk AND > ss_store_sk = s_store_sk AND time_dim.t_hour = 9 AND time_dim.t_minute < 30 > AND ((household_demographics.hd_dep_count = 4 AND > household_demographics.hd_vehicle_count <= 4 + 2) OR > (household_demographics.hd_dep_count = 2 AND > household_demographics.hd_vehicle_count <= 2 + 2) OR > (household_demographics.hd_dep_count = 0 AND > household_demographics.hd_vehicle_count <= 0 + 2)) AND store.s_store_name = > 'ese') s2, (SELECT count(*) h9_30_to_10 FROM store_sales, > household_demographics, time_dim, store WHERE ss_sold_time_sk = > time_dim.t_time_sk AND ss_hdemo_sk = household_demographics.hd_demo_sk AND > ss_store_sk = s_store_sk AND time_dim.t_hour = 9 AND time_dim.t_minute >= 30 > AND ((household_demographics.hd_dep_count = 4 AND > household_demographics.hd_vehicle_count <= 4 + 2) OR > (household_demographics.hd_dep_count = 2 AND > household_demographics.hd_vehicle_count <= 2 + 2) OR > (household_demographics.hd_dep_count = 0 AND > household_demographics.hd_vehicle_count <= 0 + 2)) AND store.s_store_name = > 'ese') s3,(SELECT count(*) h10_to_10_30 FROM store_sales, > household_demographics, time_dim, store WHERE ss_sold_time_sk = > time_dim.t_time_sk AND ss_hdemo_sk = household_demographics.hd_demo_sk AND > ss_store_sk = s_store_sk AND time_dim.t_hour = 10 AND time_dim.t_minute < 30 > AND ((household_demographics.hd_dep_count = 4 AND > household_demographics.hd_vehicle_count <= 4 + 2) OR > (household_demographics.hd_dep_count = 2 AND > household_demographics.hd_vehicle_count <= 2 + 2) OR > (household_demographics.hd_dep_count = 0 AND > household_demographics.hd_vehicle_count <= 0 + 2)) AND store.s_store_name = > 'ese') s4,(SELECT count(*) h10_30_to_11 FROM > store_sales,household_demographics, time_dim, store WHERE ss_sold_time_sk = > time_dim.t_time_sk AND ss_hdemo_sk = household_demographics.hd_demo_sk AND > ss_store_sk = s_store_sk AND time_dim.t_hour = 10 AND time_dim.t_minute >= 30 > AND ((household_demographics.hd_dep_count = 4 AND > household_demographics.hd_vehicle_count <= 4 + 2) OR > (household_demographics.hd_dep_count = 2 AND > household_demographics.hd_vehicle_count <= 2 + 2) OR > (household_demographics.hd_dep_count = 0 AND > household_demographics.hd_vehicle_count <= 0 + 2)) AND store.s_store_name = > 'ese') s5,(SELECT count(*) h11_to_11_30 FROM store_sales, > household_demographics, time_dim, store WHERE ss_sold_time_sk = > time_dim.t_time_sk AND ss_hdemo_sk = household_demographics.hd_demo_sk AND > ss_store_sk = s_store_sk AND time_dim.t_hour = 11 AND time_dim.t_minute < 30 > AND ((household_demographics.hd_dep_count = 4 AND > household_demographics.hd_vehicle_count <= 4 + 2) OR > (household_demographics.hd_dep_count = 2 AND > household_demographics.hd_vehicle_count <= 2 + 2) OR > (household_demographics.hd_dep_count = 0 AND > household_demographics.hd_vehicle_count <= 0 + 2)) AND store.s_store_name = > 'ese') s6,(SELECT count(*) h11_30_to_12 FROM store_sales, > household_demographics, time_dim, store WHERE ss_sold_time_sk = >
[jira] [Commented] (SPARK-29584) NOT NULL is not supported in Spark
[ https://issues.apache.org/jira/browse/SPARK-29584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16958540#comment-16958540 ] pavithra ramachandran commented on SPARK-29584: --- i shall work on this > NOT NULL is not supported in Spark > -- > > Key: SPARK-29584 > URL: https://issues.apache.org/jira/browse/SPARK-29584 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.0.0 >Reporter: ABHISHEK KUMAR GUPTA >Priority: Major > > Spark while creating table restricting column for NULL value is not supported. > As below > PostgreSQL: SUCCESS No Exception > CREATE TABLE Persons (ID int *NOT NULL*, LastName varchar(255) *NOT > NULL*,FirstName varchar(255) NOT NULL, Age int); > insert into Persons values(1,'GUPTA','Abhi',NULL); > select * from persons; > > Spark: Parse Exception > jdbc:hive2://10.18.19.208:23040/default> CREATE TABLE Persons (ID int NOT > NULL, LastName varchar(255) NOT NULL,FirstName varchar(255) NOT NULL, Age > int); > Error: org.apache.spark.sql.catalyst.parser.ParseException: > no viable alternative at input 'CREATE TABLE Persons (ID int NOT'(line 1, pos > 29) > Parse Exception -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29504) Tooltip not display for Job Description even it shows ellipsed
[ https://issues.apache.org/jira/browse/SPARK-29504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955807#comment-16955807 ] pavithra ramachandran commented on SPARK-29504: --- [~sandeep.katta2007] i tested on the latest on double click its not expanding. I tested in 2.3.4 on double click it is expanding. I ll check why in master it is broken and fix asap > Tooltip not display for Job Description even it shows ellipsed > --- > > Key: SPARK-29504 > URL: https://issues.apache.org/jira/browse/SPARK-29504 > Project: Spark > Issue Type: Sub-task > Components: Web UI >Affects Versions: 3.0.0 >Reporter: ABHISHEK KUMAR GUPTA >Priority: Major > Attachments: ToolTip JIRA.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29504) Tooltip not display for Job Description even it shows ellipsed
[ https://issues.apache.org/jira/browse/SPARK-29504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16954270#comment-16954270 ] pavithra ramachandran commented on SPARK-29504: --- i shall work on this > Tooltip not display for Job Description even it shows ellipsed > --- > > Key: SPARK-29504 > URL: https://issues.apache.org/jira/browse/SPARK-29504 > Project: Spark > Issue Type: Sub-task > Components: Web UI >Affects Versions: 3.0.0 >Reporter: ABHISHEK KUMAR GUPTA >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29476) Add tooltip information for Thread Dump links and Thread details table columns in Executors Tab
[ https://issues.apache.org/jira/browse/SPARK-29476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951839#comment-16951839 ] pavithra ramachandran commented on SPARK-29476: --- i shall work on this > Add tooltip information for Thread Dump links and Thread details table > columns in Executors Tab > --- > > Key: SPARK-29476 > URL: https://issues.apache.org/jira/browse/SPARK-29476 > Project: Spark > Issue Type: Sub-task > Components: Web UI >Affects Versions: 3.0.0 >Reporter: jobit mathew >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29460) Improve tooltip for Job Tab
[ https://issues.apache.org/jira/browse/SPARK-29460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950749#comment-16950749 ] pavithra ramachandran commented on SPARK-29460: --- i shall work on this > Improve tooltip for Job Tab > --- > > Key: SPARK-29460 > URL: https://issues.apache.org/jira/browse/SPARK-29460 > Project: Spark > Issue Type: Sub-task > Components: Web UI >Affects Versions: 3.0.0 >Reporter: ABHISHEK KUMAR GUPTA >Priority: Major > > [~LI,Xiao] I see there is inconsistency in the tool tip added for the column > across the tab. like Duration column in Job Tab does not have tooltip but > Duration Column in JDBC/ODBC Server tab has tooltip. > I submitted this Jira to handle this inconsistency in Job Tab Table Column. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29457) Improve tooltip information for Environment Tab
[ https://issues.apache.org/jira/browse/SPARK-29457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950729#comment-16950729 ] pavithra ramachandran commented on SPARK-29457: --- i ll work on this > Improve tooltip information for Environment Tab > --- > > Key: SPARK-29457 > URL: https://issues.apache.org/jira/browse/SPARK-29457 > Project: Spark > Issue Type: Sub-task > Components: Web UI >Affects Versions: 3.0.0 >Reporter: ABHISHEK KUMAR GUPTA >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29456) Add tooltip information for Session Statistics Table column in JDBC/ODBC Server Tab
[ https://issues.apache.org/jira/browse/SPARK-29456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950720#comment-16950720 ] pavithra ramachandran commented on SPARK-29456: --- i will work on this > Add tooltip information for Session Statistics Table column in JDBC/ODBC > Server Tab > > > Key: SPARK-29456 > URL: https://issues.apache.org/jira/browse/SPARK-29456 > Project: Spark > Issue Type: Sub-task > Components: Web UI >Affects Versions: 3.0.0 >Reporter: ABHISHEK KUMAR GUPTA >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28794) Document CREATE TABLE in SQL Reference.
[ https://issues.apache.org/jira/browse/SPARK-28794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934815#comment-16934815 ] pavithra ramachandran commented on SPARK-28794: --- Yes... I raise a PR soon.. > Document CREATE TABLE in SQL Reference. > --- > > Key: SPARK-28794 > URL: https://issues.apache.org/jira/browse/SPARK-28794 > Project: Spark > Issue Type: Sub-task > Components: Documentation, SQL >Affects Versions: 2.4.3 >Reporter: Dilip Biswal >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29108) Add window.sql - Part 2
[ https://issues.apache.org/jira/browse/SPARK-29108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931071#comment-16931071 ] pavithra ramachandran commented on SPARK-29108: --- i shall work on this > Add window.sql - Part 2 > --- > > Key: SPARK-29108 > URL: https://issues.apache.org/jira/browse/SPARK-29108 > Project: Spark > Issue Type: Sub-task > Components: SQL, Tests >Affects Versions: 3.0.0 >Reporter: Dylan Guedes >Priority: Major > Fix For: 3.0.0 > > > In this ticket, we plan to add the regression test cases of > [https://github.com/postgres/postgres/blob/REL_12_BETA3/src/test/regress/sql/window.sql#L320-L562|https://github.com/postgres/postgres/blob/REL_12_BETA3/src/test/regress/sql/window.sql#L320-L562] -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28972) [Spark] spark.memory.offHeap.size description require to update in document
[
https://issues.apache.org/jira/browse/SPARK-28972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922414#comment-16922414
]
pavithra ramachandran commented on SPARK-28972:
---
i will work on this
> [Spark] spark.memory.offHeap.size description require to update in document
> ---
>
> Key: SPARK-28972
> URL: https://issues.apache.org/jira/browse/SPARK-28972
> Project: Spark
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 2.4.3
>Reporter: ABHISHEK KUMAR GUPTA
>Priority: Minor
>
>
> spark.memory.offHeap.size accept 1G or 1KB also. So User is able to give
> suffix also but description say only *'absolute amount of memory in bytes'.*
> This require to update like *spark.driver.memory* where it is mentioned is
> accepts *a size unit suffix ("k", "m", "g" or "t") (e.g. {{512m}}, {{2g}}).*
>
> |{{spark.memory.offHeap.size}}|0|The *absolute amount of memory in bytes*
> which can be used for off-heap allocation. This setting has no impact on heap
> memory usage, so if your executors' total memory consumption must fit within
> some hard limit then be sure to shrink your JVM heap size accordingly. This
> must be set to a positive value when {{spark.memory.offHeap.enabled=true}}.|
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Issue Comment Deleted] (SPARK-28929) Spark Logging level should be INFO instead of Debug in Executor Plugin API[SPARK-24918]
[
https://issues.apache.org/jira/browse/SPARK-28929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
pavithra ramachandran updated SPARK-28929:
--
Comment: was deleted
(was: I am working on this
)
> Spark Logging level should be INFO instead of Debug in Executor Plugin
> API[SPARK-24918]
> ---
>
> Key: SPARK-28929
> URL: https://issues.apache.org/jira/browse/SPARK-28929
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.4.2, 2.4.3
>Reporter: jobit mathew
>Priority: Minor
>
> Spark Logging level should be INFO instead of Debug in Executor Plugin
> API[SPARK-24918].
> Currently logging level for Executor Plugin API[SPARK-24918] is DEBUG
> logDebug(s"Initializing the following plugins: $\{pluginNames.mkString(",
> ")}")
> logDebug(s"Successfully loaded plugin " +
> plugin.getClass().getCanonicalName())
> logDebug("Finished initializing plugins")
> It is better to change to INFO instead of DEBUG.
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28929) Spark Logging level should be INFO instead of Debug in Executor Plugin API[SPARK-24918]
[
https://issues.apache.org/jira/browse/SPARK-28929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919523#comment-16919523
]
pavithra ramachandran commented on SPARK-28929:
---
I am working on this
> Spark Logging level should be INFO instead of Debug in Executor Plugin
> API[SPARK-24918]
> ---
>
> Key: SPARK-28929
> URL: https://issues.apache.org/jira/browse/SPARK-28929
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.4.2, 2.4.3
>Reporter: jobit mathew
>Priority: Minor
>
> Spark Logging level should be INFO instead of Debug in Executor Plugin
> API[SPARK-24918].
> Currently logging level for Executor Plugin API[SPARK-24918] is DEBUG
> logDebug(s"Initializing the following plugins: $\{pluginNames.mkString(",
> ")}")
> logDebug(s"Successfully loaded plugin " +
> plugin.getClass().getCanonicalName())
> logDebug("Finished initializing plugins")
> It is better to change to INFO instead of DEBUG.
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28897) Invalid usage of '*' in expression 'coalesce' error when executing dataframe.na.fill(0)
[
https://issues.apache.org/jira/browse/SPARK-28897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16917733#comment-16917733
]
pavithra ramachandran commented on SPARK-28897:
---
i will work on this
> Invalid usage of '*' in expression 'coalesce' error when executing
> dataframe.na.fill(0)
> ---
>
> Key: SPARK-28897
> URL: https://issues.apache.org/jira/browse/SPARK-28897
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.4.0
>Reporter: Saurabh Santhosh
>Priority: Major
>
> Getting the following error when trying to execute the given statements
>
> {code:java}
> var df = spark.sql(s"select * from default.test_table")
> df.na.fill(0)
> {code}
> This error happens when the following property is set
> {code:java}
> spark.sql("set spark.sql.parser.quotedRegexColumnNames=true")
> {code}
> Error :
> {code:java}
> org.apache.spark.sql.AnalysisException: Invalid usage of '*' in expression
> 'coalesce'; at
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:42)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:95)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$expandStarExpression$1.applyOrElse(Analyzer.scala:1021)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$expandStarExpression$1.applyOrElse(Analyzer.scala:997)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:278)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:278)
> at
> org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:277)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:275)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:275)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:326)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:324)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:275)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$.expandStarExpression(Analyzer.scala:997)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveReferences$$buildExpandedProjectList$1.apply(Analyzer.scala:982)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveReferences$$buildExpandedProjectList$1.apply(Analyzer.scala:977)
> at
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
> at
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at
> scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
> at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104) at
> org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveReferences$$buildExpandedProjectList(Analyzer.scala:977)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$9.applyOrElse(Analyzer.scala:905)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$9.applyOrElse(Analyzer.scala:900)
> at
> org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1$$anonfun$apply$1.apply(AnalysisHelper.scala:90)
> at
> org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1$$anonfun$apply$1.apply(AnalysisHelper.scala:90)
> at
> org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
> at
> org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1.apply(AnalysisHelper.scala:89)
> at
> org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$$anonfun$resolveOperatorsUp$1.apply(AnalysisHelper.scala:86)
> at
> org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:194)
> at
> org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$class.resolveOperatorsUp(AnalysisHelper.scala:86)
[jira] [Commented] (SPARK-28799) Document TRUNCATE TABLE in SQL Reference.
[ https://issues.apache.org/jira/browse/SPARK-28799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912054#comment-16912054 ] pavithra ramachandran commented on SPARK-28799: --- i will work on this > Document TRUNCATE TABLE in SQL Reference. > - > > Key: SPARK-28799 > URL: https://issues.apache.org/jira/browse/SPARK-28799 > Project: Spark > Issue Type: Sub-task > Components: Documentation, SQL >Affects Versions: 2.4.3 >Reporter: Dilip Biswal >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28794) Document CREATE TABLE in SQL Reference.
[ https://issues.apache.org/jira/browse/SPARK-28794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912050#comment-16912050 ] pavithra ramachandran commented on SPARK-28794: --- i will work on this > Document CREATE TABLE in SQL Reference. > --- > > Key: SPARK-28794 > URL: https://issues.apache.org/jira/browse/SPARK-28794 > Project: Spark > Issue Type: Sub-task > Components: Documentation, SQL >Affects Versions: 2.4.3 >Reporter: Dilip Biswal >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28791) Document ALTER TABLE statement in SQL Reference.
[ https://issues.apache.org/jira/browse/SPARK-28791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912023#comment-16912023 ] pavithra ramachandran commented on SPARK-28791: --- i will work on this > Document ALTER TABLE statement in SQL Reference. > > > Key: SPARK-28791 > URL: https://issues.apache.org/jira/browse/SPARK-28791 > Project: Spark > Issue Type: Sub-task > Components: Documentation >Affects Versions: 2.4.3 >Reporter: Dilip Biswal >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28672) [UDF] Duplicate function creation should not allow
[
https://issues.apache.org/jira/browse/SPARK-28672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911003#comment-16911003
]
pavithra ramachandran commented on SPARK-28672:
---
[~maropu] [~viirya]
The intention of this Jira is to not allow creation of temporary function, when
a permanent function with the same name exist.
Could you confirm if its ok to handle this case?
> [UDF] Duplicate function creation should not allow
> ---
>
> Key: SPARK-28672
> URL: https://issues.apache.org/jira/browse/SPARK-28672
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.0
>Reporter: ABHISHEK KUMAR GUPTA
>Priority: Minor
>
> {code}
> 0: jdbc:hive2://10.18.18.214:23040/default> create function addm_3 AS
> 'com.huawei.bigdata.hive.example.udf.multiply' using jar
> 'hdfs://hacluster/user/Multiply.jar';
> +-+--+
> | Result |
> +-+--+
> +-+--+
> No rows selected (0.084 seconds)
> {code}
> {code}
> 0: jdbc:hive2://10.18.18.214:23040/default> create temporary function addm_3
> AS 'com.huawei.bigdata.hive.example.udf.multiply' using jar
> 'hdfs://hacluster/user/Multiply.jar';
> INFO : converting to local hdfs://hacluster/user/Multiply.jar
> INFO : Added
> [/tmp/8a396308-41f8-4335-9de4-8268ce5c70fe_resources/Multiply.jar] to class
> path
> INFO : Added resources: [hdfs://hacluster/user/Multiply.jar]
> +-+--+
> | Result |
> +-+--+
> +-+--+
> No rows selected (0.134 seconds)
> {code}
> {code}
> 0: jdbc:hive2://10.18.18.214:23040/default> show functions like addm_3;
> +-+--+
> |function |
> +-+--+
> | addm_3 |
> | default.addm_3 |
> +-+--+
> 2 rows selected (0.047 seconds)
> {code}
> When show function executed it is listing both the function but what about
> the db for permanent function when user has not specified.
> Duplicate should not be allowed if user creating temporary one with the same
> name.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28672) [UDF] Duplicate function creation should not allow
[
https://issues.apache.org/jira/browse/SPARK-28672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911002#comment-16911002
]
pavithra ramachandran commented on SPARK-28672:
---
[~abhishek.akg] - When we execute show function- it is displaying the
temporary and permanent function that is created, since no database is
mentioned while creating the permanent it is stored as default.addm3, Temporary
functions are not specific to any database, So it is displayed without any db
name. I dont think that is an issue
> [UDF] Duplicate function creation should not allow
> ---
>
> Key: SPARK-28672
> URL: https://issues.apache.org/jira/browse/SPARK-28672
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.0
>Reporter: ABHISHEK KUMAR GUPTA
>Priority: Minor
>
> {code}
> 0: jdbc:hive2://10.18.18.214:23040/default> create function addm_3 AS
> 'com.huawei.bigdata.hive.example.udf.multiply' using jar
> 'hdfs://hacluster/user/Multiply.jar';
> +-+--+
> | Result |
> +-+--+
> +-+--+
> No rows selected (0.084 seconds)
> {code}
> {code}
> 0: jdbc:hive2://10.18.18.214:23040/default> create temporary function addm_3
> AS 'com.huawei.bigdata.hive.example.udf.multiply' using jar
> 'hdfs://hacluster/user/Multiply.jar';
> INFO : converting to local hdfs://hacluster/user/Multiply.jar
> INFO : Added
> [/tmp/8a396308-41f8-4335-9de4-8268ce5c70fe_resources/Multiply.jar] to class
> path
> INFO : Added resources: [hdfs://hacluster/user/Multiply.jar]
> +-+--+
> | Result |
> +-+--+
> +-+--+
> No rows selected (0.134 seconds)
> {code}
> {code}
> 0: jdbc:hive2://10.18.18.214:23040/default> show functions like addm_3;
> +-+--+
> |function |
> +-+--+
> | addm_3 |
> | default.addm_3 |
> +-+--+
> 2 rows selected (0.047 seconds)
> {code}
> When show function executed it is listing both the function but what about
> the db for permanent function when user has not specified.
> Duplicate should not be allowed if user creating temporary one with the same
> name.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28653) Create table using DDL statement should not auto create the destination folder
[
https://issues.apache.org/jira/browse/SPARK-28653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903834#comment-16903834
]
pavithra ramachandran commented on SPARK-28653:
---
i will check this
> Create table using DDL statement should not auto create the destination folder
> --
>
> Key: SPARK-28653
> URL: https://issues.apache.org/jira/browse/SPARK-28653
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.4.3
>Reporter: Thanida
>Priority: Minor
>
> I create external table using this following DDL statement, the destination
> path was auto-created.
> {code:java}
> CREATE TABLE ${tableName} USING parquet LOCATION ${path}
> {code}
> But, if I specified file format as csv or json, the destination path was not
> created.
> {code:java}
> CREATE TABLE ${tableName} USING CSV LOCATION ${path}
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28672) [UDF] Duplicate function creation should not allow
[ https://issues.apache.org/jira/browse/SPARK-28672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903584#comment-16903584 ] pavithra ramachandran commented on SPARK-28672: --- i will work on this > [UDF] Duplicate function creation should not allow > --- > > Key: SPARK-28672 > URL: https://issues.apache.org/jira/browse/SPARK-28672 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: ABHISHEK KUMAR GUPTA >Priority: Minor > > 0: jdbc:hive2://10.18.18.214:23040/default> create function addm_3 AS > 'com.huawei.bigdata.hive.example.udf.multiply' using jar > 'hdfs://hacluster/user/Multiply.jar'; > +-+--+ > | Result | > +-+--+ > +-+--+ > No rows selected (0.084 seconds) > 0: jdbc:hive2://10.18.18.214:23040/default> create temporary function addm_3 > AS 'com.huawei.bigdata.hive.example.udf.multiply' using jar > 'hdfs://hacluster/user/Multiply.jar'; > INFO : converting to local hdfs://hacluster/user/Multiply.jar > INFO : Added > [/tmp/8a396308-41f8-4335-9de4-8268ce5c70fe_resources/Multiply.jar] to class > path > INFO : Added resources: [hdfs://hacluster/user/Multiply.jar] > +-+--+ > | Result | > +-+--+ > +-+--+ > No rows selected (0.134 seconds) > 0: jdbc:hive2://10.18.18.214:23040/default> show functions like addm_3; > +-+--+ > |function | > +-+--+ > | addm_3 | > | default.addm_3 | > +-+--+ > 2 rows selected (0.047 seconds) > 0: jdbc:hive2://10.18.18.214:23040/default> > When show function executed it is listing both the function but what about > the db for permanent function when user has not specified. > Duplicate should not be allowed if user creating temporary one with the same > name. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28671) [UDF] dropping permanent function when a temporary function with the same name already exists giving wrong msg on dropping it again
[ https://issues.apache.org/jira/browse/SPARK-28671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903570#comment-16903570 ] pavithra ramachandran commented on SPARK-28671: --- i will work on this > [UDF] dropping permanent function when a temporary function with the same > name already exists giving wrong msg on dropping it again > --- > > Key: SPARK-28671 > URL: https://issues.apache.org/jira/browse/SPARK-28671 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: ABHISHEK KUMAR GUPTA >Priority: Minor > > Created jar and uploaded at hdfs path > 1../hdfs dfs -put /opt/trash1/AddDoublesUDF.jar /user/user1/ > 2.Launch beeline and created permanent function > CREATE FUNCTION addDoubles AS > 'com.huawei.bigdata.hive.example.udf.AddDoublesUDF' using jar > 'hdfs://hacluster/user/user1/AddDoublesUDF.jar'; > 3.Perform select operation > jdbc:hive2://100.100.208.125:23040/default> SELECT addDoubles(1,2,3); > +--+--+ > | default.addDoubles(1, 2, 3) | > +--+--+ > | 6.0 | > +--+--+ > 1 row selected (0.111 seconds) > 4.Created temporary function as below > jdbc:hive2://100.100.208.125:23040/default> CREATE temporary FUNCTION > addDoubles AS 'com.huawei.bigdata.hive.example.udf.AddDoublesUDF' using jar > 'hdfs://hacluster/user/user1/AddDoublesUDF.jar'; > 5.jdbc:hive2://100.100.208.125:23040/default> SELECT addDoubles(1,2,3); > +--+--+ > | addDoubles(1, 2, 3) | > +--+--+ > | 6.0 | > +--+--+ > 1 row selected (0.088 seconds) > 6.Drop function > jdbc:hive2://100.100.208.125:23040/default> drop function addDoubles; > +-+--+ > | Result | > +-+--+ > +-+--+ > 7.jdbc:hive2://100.100.208.125:23040/default> SELECT addDoubles(1,2,3); > -- It is success > 8.Drop again Error thrown > jdbc:hive2://100.100.208.125:23040/default> drop function addDoubles; > Error: org.apache.spark.sql.catalyst.analysis.NoSuchFunctionException: > Undefined function: 'default.addDoubles'. This function is neither a > registered temporary function nor a permanent function registered in the > database 'default'.; (state=,code=0) > 9.Perform again select > jdbc:hive2://100.100.208.125:23040/default> SELECT addDoubles(1,2,3); > +--+--+ > | addDoubles(1, 2, 3) | > +--+--+ > | 6.0 | > > Issue is why the Error msg shown is step 8 saying it is neither registered as > permanent or temporary function where as it is registered as temporary > function in step 4 that is why in step 9 select is returning result. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28338) spark.read.format("csv") treat empty string as null if csv file don't have quotes in data
[
https://issues.apache.org/jira/browse/SPARK-28338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882872#comment-16882872
]
pavithra ramachandran commented on SPARK-28338:
---
Could you given more information, as to what is your exact expectation by
giving an example ?
> spark.read.format("csv") treat empty string as null if csv file don't have
> quotes in data
> -
>
> Key: SPARK-28338
> URL: https://issues.apache.org/jira/browse/SPARK-28338
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.3
>Reporter: Jayadevan M
>Priority: Major
>
> The csv input file
> +cat sample.csv+
> Name,Lastname,Age
> abc,,32
> pqr,xxx,30
>
> +spark-shell+
> spark.read.format("csv").option("header",
> "true").load("/media/ub_share/projects/*.csv").head(3)
> res14: Array[org.apache.spark.sql.Row] = Array([abc,null,32], [pqr,xxx,30])
>
> scala> spark.read.format("csv").option("header", "true").option("nullValue",
> "?").load("/media/ub_share/projects/*.csv").head(3)
> res15: Array[org.apache.spark.sql.Row] = Array([abc,null,32], [pqr,xxx,30])
>
> The empty string get converted to null. Its works fine if the csv file have
> quotes in columns.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
