Error While Running Merge Statement With Iceberg

Hi I am trying to run merge statement with Iceberg with Spark I am getting below error THis is the statement MERGE INTO glue.euc_dp_prd.emp_merge_test AS t USING source AS s ON t.emp_id=s.emp_id WHEN MATCHED THEN UPDATE SET t.name=s.name , t.dept=s.dept , t.doj=s.doj pyspark.errors.exceptions.

Create Custom Logs

Hi I justed wanted to check if there is a way to create custom log in Spark I want to write selective/custom log messages to S3 , running spark submit on EMR I would not want all the spark generated logs ... I would just need the log messages that are logged as part of Spark Application

Structured Streaming Process Each Records Individually

Hi I have a use case where I need to process json payloads coming from Kafka using structured streaming , but thing is json can have different formats , schema is not fixed and each json will have a @type tag so based on tag , json has to be parsed and loaded to table with tag name , and if a json

Re: Select Columns from Dataframe in Java

new Column[0])) > > > > > On Fri, Dec 29, 2023 at 10:58 PM PRASHANT L wrote: > >> >> Team >> I am using Java and want to select columns from Dataframe , columns are >> stored in List >> equivalent of below scala code >> * array_df=array_df.select(fields:

Select Columns from Dataframe in Java

Team I am using Java and want to select columns from Dataframe , columns are stored in List equivalent of below scala code * array_df=array_df.select(fields: _*)* When I try array_df=array_df.select(fields) , I get error saying Cast to Column I am using Spark 3.4