Spark SQL readSideCharPadding issue while reading ENUM column from mysql

I have upgraded my spark job from spark 3.3.1 to spark 3.5.0, I am querying to Mysql Database and applying `*UPPER(col) = UPPER(value)*` in the subsequent sql query. It is working as expected in spark 3.3.1 , but not working with 3.5.0. Where Condition :: `*UPPER(vn) = 'ERICSSON' AND (upper(st)

Re: [Issue] Spark SQL - broadcast failure

Hi all, Do we have any idea on this. Thanks On Tue, 23 Jul, 2024, 12:54 pm Sudharshan V, wrote: > We removed the explicit broadcast for that particular table and it took > longer time since the join type changed from BHJ to SMJ. > > I wanted to understand how I can find what went wrong with the

Re: [Issue] Spark SQL - broadcast failure

We removed the explicit broadcast for that particular table and it took longer time since the join type changed from BHJ to SMJ. I wanted to understand how I can find what went wrong with the broadcast now. How do I know the size of the table inside of spark memory. I have tried to cache the tabl

Re: [Issue] Spark SQL - broadcast failure

Hi all, apologies for the delayed response. We are using spark version 3.4.1 in jar and EMR 6.11 runtime. We have disabled the auto broadcast always and would broadcast the smaller tables using explicit broadcast. It was working fine historically and only now it is failing. The data sizes I men

[Spark SQL]: Why the OptimizeSkewedJoin rule does not optimize FullOuterJoin?

Hi, I am a beginner in Spark and currently learning the Spark source code. I have a question about the AQE rule OptimizeSkewedJoin. I have a SQL query using SMJ FullOuterJoin, where there is read skew on the left side (the case is mentioned below). case: remote bytes read total (min, med, max)

Re: [Issue] Spark SQL - broadcast failure

Can you try disabling broadcast join and see what happens? On Mon, Jul 8, 2024 at 12:03 PM Sudharshan V wrote: > Hi all, > > Been facing a weird issue lately. > In our production code base , we have an explicit broadcast for a small > table. > It is just a look up table that is around 1gb in siz

Re: [Issue] Spark SQL - broadcast failure

It will help if you mention the Spark version and the piece of problematic code HTH Mich Talebzadeh, Technologist | Architect | Data Engineer | Generative AI | FinCrime PhD Imperial College London

[Issue] Spark SQL - broadcast failure

Hi all, Been facing a weird issue lately. In our production code base , we have an explicit broadcast for a small table. It is just a look up table that is around 1gb in size in s3 and just had few million records and 5 columns. The ETL was running fine , but with no change from the codebase nor

Re: [Spark SQL]: Does Spark support processing records with timestamp NULL in stateful streaming?

When you use applyInPandasWithState, Spark processes each input row as it arrives, regardless of whether certain columns, such as the timestamp column, contain NULL values. This behavior is useful where you want to handle incomplete or missing data gracefully within your stateful processing logic.

[Spark SQL]: Does Spark support processing records with timestamp NULL in stateful streaming?

I am using applyInPandasWithState in PySpark 3.5.0. I noticed that records with timestamp==NULL are processed (i.e., trigger a call to the stateful function). And, as you would expect, does not advance the watermark. I am taking advantage of this in my application. My question: Is this a support

Re: Subject: [Spark SQL] [Debug] Spark Memory Issue with DataFrame Processing

rk.apache.org Subject: Re: Subject: [Spark SQL] [Debug] Spark Memory Issue with DataFrame Processing This message contains hyperlinks, take precaution before opening these links. Few ideas on top of my head for how to go about solving the problem 1. Try with subsets: Try reproducing t

Re: Subject: [Spark SQL] [Debug] Spark Memory Issue with DataFrame Processing

in Spark 2.4.0. > > > *Heap Dump Analysis:*We performed a heap dump analysis after enabling > heap dump on out-of-memory errors, and the analysis revealed the following > significant frames and local variables: > > ``` > > org.apache.spark.sql.Dataset.withPlan(Lorg/a

Subject: [Spark SQL] [Debug] Spark Memory Issue with DataFrame Processing

park 2.4.0. *Heap Dump Analysis:*We performed a heap dump analysis after enabling heap dump on out-of-memory errors, and the analysis revealed the following significant frames and local variables: ``` org.apache.spark.sql.Dataset.withPlan(Lorg/apache/spark/sql/catalyst/plans/logical/Logical

Re: Issue with Materialized Views in Spark SQL

nformation provided is correct to the best of my > knowledge but of course cannot be guaranteed . It is essential to note > that, as with any advice, quote "one test result is worth one-thousand > expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von

Re: Issue with Materialized Views in Spark SQL

ny advice, quote "one test result is worth one-thousand expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". On Fri, 3 May 2024 at 00:54, Mich Talebzadeh wrote: > An issue I encountered while

Re: Issue with Materialized Views in Spark SQL

not a bug or an issue. You can initiate a feature request and wish the community to include that into the roadmap. On Fri, May 3, 2024 at 12:01 PM Mich Talebzadeh wrote: > An issue I encountered while working with Materialized Views in Spark SQL. > It appears that there is an inconsistency be

Re: Issue with Materialized Views in Spark SQL

Thanks, Walaa. On Thu, May 2, 2024 at 4:55 PM Mich Talebzadeh wrote: > An issue I encountered while working with Materialized Views in Spark SQL. > It appears that there is an inconsistency between the behavior of > Materialized Views in Spark SQL and Hive. > > When attempti

Issue with Materialized Views in Spark SQL

An issue I encountered while working with Materialized Views in Spark SQL. It appears that there is an inconsistency between the behavior of Materialized Views in Spark SQL and Hive. When attempting to execute a statement like DROP MATERIALIZED VIEW IF EXISTS test.mv in Spark SQL, I encountered a

How to use Structured Streaming in Spark SQL

In Flink, you can create flow calculation tables using Flink SQL, and directly connect with SQL through CDC and Kafka. How to use SQL for flow calculation in Spark 308027...@qq.com  

[Spark SQL][How-To] Remove builtin function support from Spark

Hello, I'm very new to the Spark ecosystem, apologies if this question is a bit simple. I want to modify a custom fork of Spark to remove function support. For example, I want to remove the query runners ability to call reflect and java_method. I saw that there exists a data structure in

[Spark SQL][How-To] Remove builtin function support from Spark

Hello, I'm very new to the Spark ecosystem, apologies if this question is a bit simple. I want to modify a custom fork of Spark to remove function support. For example, I want to remove the query runners ability to call reflect and java_method. I saw that there exists a data structure in

[Spark SQL] xxhash64 default seed of 42 confusion

Hi all, I've noticed that spark's xxhas64 output doesn't match other tool's due to using seed=42 as a default. I've looked at a few libraries and they use 0 as a default seed: - python https://github.com/ifduyue/python-xxhash - java https://github.com/OpenHFT/Zero-Allocation-Hashing/ - java (slic

Re: [Spark SQL]: Source code for PartitionedFile

Hi Mich, Thanks for the reply. I did come across that file but it didn't align with the appearance of `PartitionedFile`: https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/PartitionedFileUtil.scala In fact, the code snippet you shared

Re: Re: [Spark SQL] How can I use .sql() in conjunction with watermarks?

interesting. So below should be the corrected code with the suggestion in the [SPARK-47718] .sql() does not recognize watermark defined upstream - ASF JIRA (apache.org) # Define schema for parsing Kafka messages schema = StructType([ StructFi

Re: Re: [Spark SQL] How can I use .sql() in conjunction with watermarks?

Sorry this is not a bug but essentially a user error. Spark throws a really confusing error and I'm also confused. Please see the reply in the ticket for how to make things correct. https://issues.apache.org/jira/browse/SPARK-47718 刘唯 于2024年4月6日周六 11:41写道： > This indeed looks like a bug. I will

Re: [Spark SQL]: Source code for PartitionedFile

Hi, I believe this is the package https://raw.githubusercontent.com/apache/spark/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FilePartition.scala And the code case class FilePartition(index: Int, files: Array[PartitionedFile]) extends Partition with

[Spark SQL]: Source code for PartitionedFile

Hi All, I've been diving into the source code to get a better understanding of how file splitting works from a user perspective. I've hit a deadend at `PartitionedFile`, for which I cannot seem to find a definition? It appears though it should be found at org.apache.spark.sql.execution.datasources

Re: Re: [Spark SQL] How can I use .sql() in conjunction with watermarks?

This indeed looks like a bug. I will take some time to look into it. Mich Talebzadeh 于2024年4月3日周三 01:55写道： > > hm. you are getting below > > AnalysisException: Append output mode not supported when there are > streaming aggregations on streaming DataFrames/DataSets without watermark; > > The pro

Re: Re: [Spark SQL] How can I use .sql() in conjunction with watermarks?

hm. you are getting below AnalysisException: Append output mode not supported when there are streaming aggregations on streaming DataFrames/DataSets without watermark; The problem seems to be that you are using the append output mode when writing the streaming query results to Kafka. This mode is

RE: Re: [Spark SQL] How can I use .sql() in conjunction with watermarks?

Hi Mich, Thank you so much for your response. I really appreciate your help! You mentioned "defining the watermark using the withWatermark function on the streaming_df before creating the temporary view” - I believe this is what I’m doing and it’s not working for me. Here is the exact code snip

Re: [Spark SQL] How can I use .sql() in conjunction with watermarks?

ok let us take it for a test. The original code of mine def fetch_data(self): self.sc.setLogLevel("ERROR") schema = StructType() \ .add("rowkey", StringType()) \ .add("timestamp", TimestampType()) \ .add("temperature", IntegerType()) checkpoi

[Spark SQL] How can I use .sql() in conjunction with watermarks?

Hello! I am attempting to write a streaming pipeline that would consume data from a Kafka source, manipulate the data, and then write results to a downstream sink (Kafka, Redis, etc). I want to write fully formed SQL instead of using the function API that Spark offers. I read a few guides on ho

[Spark SQL]: Crash when attempting to select PostgreSQL bpchar without length specifier in Spark 3.5.0

Hi, I’m currently migrating an ETL project to Spark 3.5.0 from 3.2.1 and ran into an issue with some of our queries that read from PostgreSQL databases. Any attempt to run a Spark SQL query that selects a bpchar without a length specifier from the source DB seems to crash

Re: Validate spark sql

Dear friend, thanks a ton was looking for linting for SQL for a long time, looks like https://sqlfluff.com/ is something that can be used :) Thank you so much, and wish you all a wonderful new year. Regards, Gourav On Tue, Dec 26, 2023 at 4:42 AM Bjørn Jørgensen wrote: > You can try sqlfluff

Re: Validate spark sql

> 抄 送：Nicholas Chammas; user< > user@spark.apache.org> > 主 题：Re: Validate spark sql > > Thanks Mich, Nicholas. I tried looking over the stack overflow post and > none of them > Seems to cover the syntax validation. Do you know if it's even possible to > do synta

Re: Validate spark sql

You can try sqlfluff it's a linter for SQL code and it seems to have support for sparksql man. 25. des. 2023 kl. 17:13 skrev ram manickam : > Thanks Mich, Nicholas. I tried looking over the stack overflow post and > none of them > Seem

Re: Validate spark sql

Mailing lists For broad, opinion based, ask for external resources, debug issues, bugs, contributing to the project, and scenarios, it is recommended you use the user@spark.apache.org mailing list. - user@spark.apache.org is for usa

回复：Validate spark sql

s://www.tianlang.tech/ > -- 发件人：ram manickam 发送时间：2023年12月25日(星期一) 12:58 收件人：Mich Talebzadeh 抄　送：Nicholas Chammas; user 主　题：Re: Validate spark sql Thanks Mich, Nicholas. I tried looking over the stack overflow post and none of them Seems to cov

Re: Validate spark sql

Thanks Mich, Nicholas. I tried looking over the stack overflow post and none of them Seems to cover the syntax validation. Do you know if it's even possible to do syntax validation in spark? Thanks Ram On Sun, Dec 24, 2023 at 12:49 PM Mich Talebzadeh wrote: > Well not to put too finer point on

Re: Validate spark sql

Well not to put too finer point on it, in a public forum, one ought to respect the importance of open communication. Everyone has the right to ask questions, seek information, and engage in discussions without facing unnecessary patronization. Mich Talebzadeh, Dad | Technologist | Solutions Arch

Re: Validate spark sql

This is a user-list question, not a dev-list question. Moving this conversation to the user list and BCC-ing the dev list. Also, this statement > We are not validating against table or column existence. is not correct. When you call spark.sql(…), Spark will lookup the table references and fail

[Spark-sql 3.2.4] Wrong Statistic INFO From 'ANALYZE TABLE' Command

Hi, all The ANALYZE TABLE command run from Spark on a Hive table. Question: Before I run ANALYZE TABLE' Command on Spark-sql client, I ran 'ANALYZE TABLE' Command on Hive client, the wrong Statistic Info show up. For example 1. run the analyze table command o hive client

Re: [ SPARK SQL ]: UPPER in WHERE condition is not working in Apache Spark 3.5.0 for Mysql ENUM Column

Any update on this? On Fri, 13 Oct, 2023, 12:56 pm Suyash Ajmera, wrote: > This issue is related to CharVarcharCodegenUtils readSidePadding method . > > Appending white spaces while reading ENUM data from mysql > > Causing issue in querying , writing the same data to Cassandra. > > On Thu, 12 O

[Spark SQL] [Bug] Adding `checkpoint()` causes "column [...] cannot be resolved" error

Hi all, Wondering if anyone has run into this as I can't find any similar issues in JIRA, mailing list archives, Stack Overflow, etc. I had a query that was running successfully, but the query planning time was extremely long (4+ hours). To fix this I added `checkpoint()` calls earlier in the code

Re: [ SPARK SQL ]: UPPER in WHERE condition is not working in Apache Spark 3.5.0 for Mysql ENUM Column

This issue is related to CharVarcharCodegenUtils readSidePadding method . Appending white spaces while reading ENUM data from mysql Causing issue in querying , writing the same data to Cassandra. On Thu, 12 Oct, 2023, 7:46 pm Suyash Ajmera, wrote: > I have upgraded my spark job from spark 3.3.

[ SPARK SQL ]: PPER in WHERE condition is not working in Apache Spark 3.5.0 for Mysql ENUM Column

I have upgraded my spark job from spark 3.3.1 to spark 3.5.0, I am querying to Mysql Database and applying `*UPPER(col) = UPPER(value)*` in the subsequent sql query. It is working as expected in spark 3.3.1 , but not working with 3.5.0. Where Condition :: `*UPPER(vn) = 'ERICSSON' AND (upper(st)

Re: Spark-SQL - Query Hanging, How To Troubleshoot

> *Disclaimer:* Use it at your own risk. Any and all >>>>>>>>>> responsibility for any loss, damage or destruction of data or any >>>>>>>>>> other >>>>>>>>>> property which may arise from relying on this ema

Re: Spark-SQL - Query Hanging, How To Troubleshoot

> Unfortunately after installing Delta Lake and re-writing all >>>>>>>>>> tables as Delta tables, the issue persists. >>>>>>>>>> >>>>>>>>>> On Sat, Aug 12, 2023 at 11:34 AM Mich Talebzadeh < >>>

Re: Spark-SQL - Query Hanging, How To Troubleshoot

erybodywiki.com/Mich_Talebzadeh >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *Disclaimer:* Use it at your own risk. Any and all >>>>>>>>>> responsibility

Re: Spark-SQL - Query Hanging, How To Troubleshoot

gt;>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Sat, 12 Aug 2023 at 12:03, Patrick Tucci < >>>>>>>>> patrick.tu...@gmail.com> wrote: >>>>>>>>> >>>

Re: Spark-SQL - Query Hanging, How To Troubleshoot

gt;>>> to Delta Lake and see if that solves the issue. >>>>>>>>> >>>>>>>>> Thanks again for your feedback. >>>>>>>>> >>>>>>>>> Patrick >>>>>>>>> >>&

Re: Spark-SQL - Query Hanging, How To Troubleshoot

;>>> columnar implementations of relational model. What you are seeing is >>>>>>>>> the >>>>>>>>> Spark API to Hive which prefers Parquet. I found out a few years ago. >>>>>>>>> >>>>>&

Re: Spark-SQL - Query Hanging, How To Troubleshoot

cing using moving to Delta Lake. >>>>>>>> >>>>>>>> You can also use compression >>>>>>>> >>>>>>>> STORED AS PARQUET >>>>>>>> TBLPROPERTIES ("parquet.compression"="SNA

Re: Spark-SQL - Query Hanging, How To Troubleshoot

nited Kingdom >>>>>>> >>>>>>> >>>>>>>view my Linkedin profile >>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>> >>>>>>> >>>>&

Re: Spark-SQL - Query Hanging, How To Troubleshoot

ta or any other property which >>>>>> may >>>>>> arise from relying on this email's technical content is explicitly >>>>>> disclaimed. The author will in no case be liable for any monetary damages >>>>>> arising from such

Re: Spark-SQL - Query Hanging, How To Troubleshoot

>>>>> >>>>> >>>>> On Fri, 11 Aug 2023 at 11:26, Patrick Tucci >>>>> wrote: >>>>> >>>>>> Thanks for the reply Stephen and Mich. >>>>>> >>>>>> Stephen, you're r

Re: Spark-SQL - Query Hanging, How To Troubleshoot

was lingering in >>>>> the background. >>>>> >>>>> Mich, thank you so much, your suggestion worked. Storing the tables as >>>>> Parquet solves the issue. >>>>> >>>>> Interestingly, I found that only the MemberEnrollm

Re: Spark-SQL - Query Hanging, How To Troubleshoot

toring the tables as >>>> Parquet solves the issue. >>>> >>>> Interestingly, I found that only the MemberEnrollment table needs to be >>>> Parquet. The ID field in MemberEnrollment is an int calculated during load >>>> by a ROW_NUMBER() functio

Re: Spark-SQL - Query Hanging, How To Troubleshoot

as MemberEnrollment.ID instead of using the ROW_NUMBER() function, the >>> query works without issue even if both tables are ORC. >>> >>> Should I infer from this issue that the Hive components prefer Parquet >>> over ORC? Furthermore, should I consider using a

Re: Spark-SQL - Query Hanging, How To Troubleshoot

ve, I'm starting to think a >> different solution might be more robust and stable. The main condition is >> that my application operates solely through Thrift server, so I need to be >> able to connect to Spark through Thrift server and have it write tables >>

Re: Spark-SQL - Query Hanging, How To Troubleshoot

rver, so I need to be > able to connect to Spark through Thrift server and have it write tables > using Delta Lake instead of Hive. From this StackOverflow question, it > looks like this is possible: > https://stackoverflow.com/questions/69862388/how-to-run-spark-sql-thrift-server-in-local-m

Re: Spark-SQL - Query Hanging, How To Troubleshoot

ely through Thrift server, so I need to be able to connect to Spark through Thrift server and have it write tables using Delta Lake instead of Hive. From this StackOverflow question, it looks like this is possible: https://stackoverflow.com/questions/69862388/how-to-run-spark-sql-thrift-server-in-

Re: Spark-SQL - Query Hanging, How To Troubleshoot

Steve may have a valid point. You raised an issue with concurrent writes before, if I recall correctly. Since this limitation may be due to Hive metastore. By default Spark uses Apache Derby for its database persistence. *However it is limited to only one Spark session at any time for the purposes

Re: Spark-SQL - Query Hanging, How To Troubleshoot

Hi Patrick, When this has happened to me in the past (admittedly via spark-submit) it has been because another job was still running and had already claimed some of the resources (cores and memory). I think this can also happen if your configuration tries to claim resources that will never be

Re: Spark-SQL - Query Hanging, How To Troubleshoot

Hi Mich, I don't believe Hive is installed. I set up this cluster from scratch. I installed Hadoop and Spark by downloading them from their project websites. If Hive isn't bundled with Hadoop or Spark, I don't believe I have it. I'm running the Thrift server distributed with Spark, like so: ~/spa

Re: Spark-SQL - Query Hanging, How To Troubleshoot

sorry host is 10.0.50.1 Mich Talebzadeh, Solutions Architect/Engineering Lead London United Kingdom view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all respons

Re: Spark-SQL - Query Hanging, How To Troubleshoot

Hi Patrick That beeline on port 1 is a hive thrift server running on your hive on host 10.0.50.1:1. if you can access that host, you should be able to log into hive by typing hive. The os user is hadoop in your case and sounds like there is no password! Once inside that host, hive logs a

Re: Spark-SQL - Query Hanging, How To Troubleshoot

Hi Mich, Thanks for the reply. Unfortunately I don't have Hive set up on my cluster. I can explore this if there are no other ways to troubleshoot. I'm using beeline to run commands against the Thrift server. Here's the command I use: ~/spark/bin/beeline -u jdbc:hive2://10.0.50.1:1 -n hadoop

Re: Spark-SQL - Query Hanging, How To Troubleshoot

Can you run this sql query through hive itself? Are you using this command or similar for your thrift server? beeline -u jdbc:hive2:///1/default org.apache.hive.jdbc.HiveDriver -n hadoop -p xxx HTH Mich Talebzadeh, Solutions Architect/Engineering Lead London United Kingdom view my Link

Spark-SQL - Query Hanging, How To Troubleshoot

Hello, I'm attempting to run a query on Spark 3.4.0 through the Spark ThriftServer. The cluster has 64 cores, 250GB RAM, and operates in standalone mode using HDFS for storage. The query is as follows: SELECT ME.*, MB.BenefitID FROM MemberEnrollment ME JOIN MemberBenefits MB ON ME.ID = MB.Enroll

Re: Spark-SQL - Concurrent Inserts Into Same Table Throws Exception

gt; HDFS by utilizing an open table format with concurrency control. Several >> formats, such as Apache Hudi, Apache Iceberg, Delta Lake, and Qbeast >> Format, offer this capability. All of them provide advanced features that >> will work better in different use cases according t

Re: Spark-SQL - Concurrent Inserts Into Same Table Throws Exception

4:28 PM Mich Talebzadeh > wrote: > >> It is not Spark SQL that throws the error. It is the underlying Database >> or layer that throws the error. >> >> Spark acts as an ETL tool. What is the underlying DB where the table >> resides? Is concurrency supported.

Re: Spark-SQL - Concurrent Inserts Into Same Table Throws Exception

that will work better in different use cases according to the writing pattern, type of queries, data characteristics, etc. *Pol Santamaria* On Sat, Jul 29, 2023 at 4:28 PM Mich Talebzadeh wrote: > It is not Spark SQL that throws the error. It is the underlying Database > or layer that

Re: Spark-SQL - Concurrent Inserts Into Same Table Throws Exception

It is not Spark SQL that throws the error. It is the underlying Database or layer that throws the error. Spark acts as an ETL tool. What is the underlying DB where the table resides? Is concurrency supported. Please send the error to this list HTH Mich Talebzadeh, Solutions Architect

Spark-SQL - Concurrent Inserts Into Same Table Throws Exception

Hello, I'm building an application on Spark SQL. The cluster is set up in standalone mode with HDFS as storage. The only Spark application running is the Spark Thrift Server using FAIR scheduling mode. Queries are submitted to Thrift Server using beeline. I have multiple queries that insert

Re: [Spark SQL] Data objects from query history

e. > > I have been exploring the capabilities of Spark SQL and Databricks, and I > have encountered a challenge related to accessing the data objects used by > queries from the query history. I am aware that Databricks provides a > comprehensive query history that contains valuable inf

[Spark SQL] Data objects from query history

exploring the capabilities of Spark SQL and Databricks, and I have encountered a challenge related to accessing the data objects used by queries from the query history. I am aware that Databricks provides a comprehensive query history that contains valuable information about executed queries. However

[Spark-SQL] Dataframe write saveAsTable failed

Hi, We have upgraded Spark from 2.4.x to 3.3.1 recently and managed table creation while writing dataframe as saveAsTable failed with below error. Can not create the managed table(``) The associated location('hdfs:') already exists. On high level our code does below before writing dataframe as t

Re: Spark-Sql - Slow Performance With CTAS and Large Gzipped File

we need the process to > go faster. > > Patrick > > On Mon, Jun 26, 2023 at 12:52 PM Mich Talebzadeh < > mich.talebza...@gmail.com> wrote: > >> OK for now have you analyzed statistics in Hive external table >> >> spark-sql (default)> ANALYZE TABLE

Re: Spark-Sql - Slow Performance With CTAS and Large Gzipped File

wrote: > OK for now have you analyzed statistics in Hive external table > > spark-sql (default)> ANALYZE TABLE test.stg_t2 COMPUTE STATISTICS FOR ALL > COLUMNS; > spark-sql (default)> DESC EXTENDED test.stg_t2; > > Hive external tables have little optimization > > HT

Re: Spark-Sql - Slow Performance With CTAS and Large Gzipped File

OK for now have you analyzed statistics in Hive external table spark-sql (default)> ANALYZE TABLE test.stg_t2 COMPUTE STATISTICS FOR ALL COLUMNS; spark-sql (default)> DESC EXTENDED test.stg_t2; Hive external tables have little optimization HTH Mich Talebzadeh, Solutions Architect/Engin

Spark-Sql - Slow Performance With CTAS and Large Gzipped File

Hello, I'm using Spark 3.4.0 in standalone mode with Hadoop 3.3.5. The master node has 2 cores and 8GB of RAM. There is a single worker node with 8 cores and 64GB of RAM. I'm trying to process a large pipe delimited file that has been compressed with gzip (9.2GB zipped, ~58GB unzipped, ~241m reco

RE: Can Spark SQL (not DataFrame or Dataset) aggregate array into map of element of count?

> > For some reason, I can ONLY do this in Spark SQL, instead of either Scala or > PySpark environment. > > I want to aggregate an array into a Map of element count, within that array, > but in Spark SQL. > I know that there is an aggregate function available like > > ag

Re: Can Spark SQL (not DataFrame or Dataset) aggregate array into map of element of count?

a'), map(), (acc, x) -> ???, acc -> acc) AS feq_cnt Here are my questions: * Is using "map()" above the best way? The "start" structure in this case should be Map.empty[String, Int], but of course, it won

Re: Can Spark SQL (not DataFrame or Dataset) aggregate array into map of element of count?

map(), (acc, x) -> ???, acc -> acc) AS feq_cnt Here are my questions: * Is using "map()" above the best way? The "start" structure in this case should be Map.empty[String, Int], but of course, it won't wor

Re: Can Spark SQL (not DataFrame or Dataset) aggregate array into map of element of count?

7;s technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Fri, 5 May 2023 at 20:33, Yong Zhang wrote: > Hi, This is on Spark 3.1 environment. > > For some reason, I can ONLY do this in Spa

Can Spark SQL (not DataFrame or Dataset) aggregate array into map of element of count?

Hi, This is on Spark 3.1 environment. For some reason, I can ONLY do this in Spark SQL, instead of either Scala or PySpark environment. I want to aggregate an array into a Map of element count, within that array, but in Spark SQL. I know that there is an aggregate function available like

Re:Upgrading from Spark SQL 3.2 to 3.3 faild

e.scala:176) at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584) at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPl

Upgrading from Spark SQL 3.2 to 3.3 faild

org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584) at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30) at

Fwd: [Spark SQL] : Delete is only supported on V2 tables.

-- Forwarded message - From: Jeevan Chhajed Date: Tue, 7 Feb 2023, 15:16 Subject: [Spark SQL] : Delete is only supported on V2 tables. To: Hi, How do we create V2 tables? I tried a couple of things using sql but was unable to do so. Can you share links/content it will be of

[Spark SQL]: Spark 3.2 generates different results to query when columns name have mixed casing vs when they have same casing

Hi Team, I am running a query in Spark 3.2. val df1 = sc.parallelize(List((1,2,3,4,5),(1,2,3,4,5))).toDF("id","col2","col3","col4", "col5") val op_cols_same_case = List("id","col2","col3","col4", "col5", "id") val df2 = df1.select(op_cols_same_case .head, op_cols_same_case .tail: _*) df2.select("

[Spark SQL] : Delete is only supported on V2 tables.

Hi, How do we create V2 tables? I tried a couple of things using sql but was unable to do so. Can you share links/content it will be of much help. Is delete support on V2 tables still under dev ? Thanks, Jeevan

SQL GROUP BY alias with dots, was: Spark SQL question

Hi, you are right, that is an interesting question. Looks like GROUP BY is doing something funny / magic here (spark-shell 3.3.1 and 3.5.0-SNAPSHOT): With an alias, it behaves as you have pointed out: spark.range(3).createTempView("ids_without_dots") spark.sql("SELECT * FROM ids_without_dots

Re: Spark SQL question

. 28. jan. 2023 kl. 09:22 skrev Mich Talebzadeh < mich.talebza...@gmail.com>: > LOL > > First one > > spark-sql> select 1 as `data.group` from abc group by data.group; > 1 > Time taken: 0.198 seconds, Fetched 1 row(s) > > means that are assigning alias data.gr

Re: Spark SQL question

LOL First one spark-sql> select 1 as `data.group` from abc group by data.group; 1 Time taken: 0.198 seconds, Fetched 1 row(s) means that are assigning alias data.group to select and you are using that alias -> data.group in your group by statement This is equivalent to spark-sql>

Spark SQL question

this SQL works select 1 as *`data.group`* from tbl group by *data.group* Since there's no such field as *data,* I thought the SQL has to look like this select 1 as *`data.group`* from tbl group by `*data.group`* But that gives and error (cannot resolve '`data.group`') ... I'm no expert in SQ

[Spark SQL] Data duplicate or data lost with non-deterministic function

Hi All, I met one data duplicate issue when writing table with shuffle data and non-deterministic function. For example: insert overwrite table target_table partition(ds) select ... from a join b join c... ditributed by ds, cast(rand()*10 as int) As rand() is non deterministic, the order of inp

Re: [Spark SQL]: unpredictable errors: java.io.IOException: can not read class org.apache.parquet.format.PageHeader

We’ve discovered a workaround for this; it’s described here<https://issues.apache.org/jira/browse/HADOOP-18521>. From: Eric Hanchrow Date: Thursday, December 8, 2022 at 17:03 To: user@spark.apache.org Subject: [Spark SQL]: unpredictable errors: java.io.IOException: can not read

[Spark SQL]: unpredictable errors: java.io.IOException: can not read class org.apache.parquet.format.PageHeader

My company runs java code that uses Spark to read from, and write to, Azure Blob storage. This code runs more or less 24x7. Recently we've noticed a few failures that leave stack traces in our logs; what they have in common are exceptions that look variously like Caused by: java.io.IOExcep

RE: Re: [Spark Sql] Global Setting for Case-Insensitive String Compare

Thanks. How would I go about formally submitting a feature request for this? On 2022/11/21 23:47:16 Andrew Melo wrote: > I think this is the right place, just a hard question :) As far as I > know, there's no "case insensitive flag", so YMMV > > On Mon, Nov 21, 2022 at 5:40 PM Patrick Tucci wrot

  1   2   3   4   5   6   7   8   9   10   >