Re: Column names returned as rows and other jdbc weirdness

One more thing that I find useful is to use a tool like Squirre l.client. It is basic but very powerful to test JDBC connections. SQuirreL SQL Client Home Page (sourceforge.net) Just install it on a node in

Re: Column names returned as rows and other jdbc weirdness

Thanks, Mich, Yes, I’m coming to the conclusion that there is a driver issue. Our ops guy is back next Monday, so I’ll see what he can do. I tried things like adding a WHERE clause so that I was sure that the row received had no nulls, in case that was the problem, but it was exactly the same.Regar

Re: Column names returned as rows and other jdbc weirdness

thanks for the info Alun. >From my experience these issues of messed up columns through JDBC are usually related to the JDBC driver being used. Your database is MariaDB. I have no direct experience of this database but it is akin to MySQL. Case in point, I had all sorts of issues connecting to Hi

Re: Column names returned as rows and other jdbc weirdness

Many thanks for the reply, Mich. I’m running Databricks on Azure. Databricks runtime version: 8.1 ML (includes Apache Spark 3.1.1, Scala 2.12)The UUID columns,  I believe, comes from Azure IoT. It is generally 36 characters, like '934c1f58-ed11-4e48-b157-aab869d9b325’, although I note some are shor

Re: Column names returned as rows and other jdbc weirdness

Hi, Which version of Spark are you using? UUID is generated by DB through OS low level call and it is 36 characters UUID=$(uuidgen) echo $UUID ef080790-4c3f-4a5f-8db7-1024338d34f2 in other words string will do it or VARCHAR(36) When you run that SQL directly on the database itself what do you

Re: Loading Hadoop-Azure in Kubernetes

Hi Nick, You should look which spark version is "latest", understand which Hadoop version was built "spark:latest" on top, and then check the compatibility of Hadoop with the Azure libraries. In the past, I used the following Dockerfile to experiment: FROM gcr.io/spark-operator/spark:v3.0.0 USER

Loading Hadoop-Azure in Kubernetes

Hello I am trying to load the Hadoop-Azure driver in Apache Spark, but so far I have failed. The plan is to include the required files in the docker image, as we plan on using a Client-mode SparkSession. My current Dockerfile looks like this: FROM spark:latest

Column names returned as rows and other jdbc weirdness

I’m just starting using PySpark (Databricks) for a education application. Part of this is monitoring children’s online behaviour to alert teachers whether there may be problems with bullying, extreme diets, suicide ideation, and so on. I have IoT data which I need to combine with information fro