Thanks for reaching out about this. It does look like a bug somewhere. I'm curious whether it works in Spark 3.5 if you have a chance to try it out.
It looks like the cause is a missing cast or incorrect schema somewhere in Spark. The data coming in is a Spark string, but for some reason Spark thinks that it should be a long. Another oddity is that Spark reports that the schema has an int for the yyyymmdd column. I think it is a good sign that the insert works. That means that normal type coercion is working for CTAS and narrows the problem to the MERGE path. Can you try adding an explicit cast in your DataFrame before running the merge? I think that might fix it. On Thu, Dec 14, 2023 at 8:02 AM Sabyasachi Nandy <sabyasachi.na...@rapido.bike> wrote: > Hey Folks, > > My team members and me have been stuck in this issue for a long time. > Can you please check this issue once. > With the snippet given in the following question - > https://stackoverflow.com/questions/77655115/org-apache-spark-unsafe-types-utf8string-cannot-be-cast-to-java-lang-long-except > you > should be able to replicate the issue. > > We will be happy to help with any further details. > > > With regards, > Sabyasachi > > THIS EMAIL COMMUNICATION IS PRIVILEGED AND MAY CONTAIN CONFIDENTIAL > INFORMATION OF RAPIDO. IF YOU ARE NOT THE INTENDED RECIPIENT, YOU ARE > HEREBY NOTIFIED THAT YOU HAVE RECEIVED THIS MESSAGE IN ERROR AND ANY > REVIEW, DISSEMINATION, DISTRIBUTION OR COPYING OF THIS MESSAGE IS STRICTLY > PROHIBITED. PLEASE NOTIFY US IMMEDIATELY BY EMAIL AND DELETE THE MESSAGE > FROM YOUR SYSTEM. > > NOTHING CONTAINED IN THIS DISCLAIMER SHALL BE CONSTRUED IN ANY WAY TO > GRANT PERMISSION TO TRANSMIT CONFIDENTIAL INFORMATION OR AS A WAIVER OF ANY > CONFIDENTIALITY OR PRIVILEGE. > > RAPIDO DOES NOT ACCEPT ANY RESPONSIBILITY OR LIABILITY ARISING FROM THE > USE OF THIS COMMUNICATION. NO REPRESENTATION IS BEING MADE THAT THE > INFORMATION PRESENTED IS ACCURATE, CURRENT OR COMPLETE AND SUCH INFORMATION > IS AT ALL TIMES SUBJECT TO CHANGE WITHOUT NOTICE > -- Ryan Blue Tabular
