Hi David, Thanks for the bug report and the proposed patch. Please file a JIRA and we'll take the discussion there.
Josh On Mon, Nov 30, 2015 at 1:01 PM, Dawid Wysakowicz < wysakowicz.da...@gmail.com> wrote: > Hi, > > I've recently found some behaviour that I found buggy when working with > phoenix-spark and arrays. > > Take a look at those unit tests: > > test("Can save arrays from custom dataframes back to phoenix") { > val dataSet = List(Row(2L, Array("String1", "String2", "String3"))) > > val sqlContext = new SQLContext(sc) > > val schema = StructType( > Seq(StructField("ID", LongType, nullable = false), > StructField("VCARRAY", ArrayType(StringType)))) > > val rowRDD = sc.parallelize(dataSet) > > // Apply the schema to the RDD. > val df = sqlContext.createDataFrame(rowRDD, schema) > > df.write > .format("org.apache.phoenix.spark") > .options(Map("table" -> "ARRAY_TEST_TABLE", "zkUrl" -> > quorumAddress)) > .mode(SaveMode.Overwrite) > .save() > } > > test("Can save arrays of AnyVal type back to phoenix") { > val dataSet = List((2L, Array(1, 2, 3), Array(1L, 2L, 3L))) > > sc > .parallelize(dataSet) > .saveToPhoenix( > "ARRAY_ANYVAL_TEST_TABLE", > Seq("ID", "INTARRAY", "BIGINTARRAY"), > zkUrl = Some(quorumAddress) > ) > > // Load the results back > val stmt = conn.createStatement() > val rs = stmt.executeQuery("SELECT INTARRAY, BIGINTARRAY FROM > ARRAY_ANYVAL_TEST_TABLE WHERE ID = 2") > rs.next() > val intArray = rs.getArray(1).getArray().asInstanceOf[Array[Int]] > val longArray = rs.getArray(2).getArray().asInstanceOf[Array[Long]] > > // Verify the arrays are equal > intArray shouldEqual dataSet(0)._2 > longArray shouldEqual dataSet(0)._3 > } > > Both fail with some ClassCastExceptions. > > In attached patch I've proposed a solution. The tricky part is with > Array[Byte] as this would be same for both VARBINARY and TINYINT[]. > > Let me know If I should create an issue for this, and if my solution > satisfies you. > > Regards > Dawid Wysakowicz > > >