Ravi Shankar created ARROW-11062:
------------------------------------
Summary: When writing to flight stream, Spark's mapPartitions is
not working
Key: ARROW-11062
URL: https://issues.apache.org/jira/browse/ARROW-11062
Project: Apache Arrow
Issue Type: Bug
Components: Java
Affects Versions: 2.0.0
Reporter: Ravi Shankar
Fix For: 2.0.0
Hi,
I have the following method:
val outRDD = myRdd.mapPartitions { it =>
val l = Location.forGrpcInsecure("10.0.0.113", 12233);
val allocator = it.allocator.newChildAllocator("SparkFlightConnector", 0,
Long.MaxValue)
val client = FlightClient.builder(allocator, l).build();
val desc = FlightDescriptor.path("wonderful")
val stream = client.startPut(desc,it.root, new AsyncPutListener)
it.foreach { root =>
// doPut on the populated VectorSchemaRoot
stream.putNext()
}
stream.completed()
// Need to call this, or exceptions from the server get swallowed
stream.getResult
// println(it.next().contentToTSVString())
client.close()
Iterator.empty
}.count
Following is the error:
Caused by: java.lang.NoSuchMethodError:
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;CLjava/lang/Object;)V
at io.grpc.Metadata$Key.validateName(Metadata.java:742)
at io.grpc.Metadata$Key.<init>(Metadata.java:750)
at io.grpc.Metadata$Key.<init>(Metadata.java:668)
at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:959)
at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:954)
at io.grpc.Metadata$Key.of(Metadata.java:705)
at io.grpc.Metadata$Key.of(Metadata.java:701)
at io.grpc.internal.GrpcUtil.<clinit>(GrpcUtil.java:80)
When I googles, its sayong some guava related jar issue. I did maven dependency
tree and did not find anything wrong. Please help.
Best,
Ravion
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
