Hello!
While operating the JavaDStream you may use a transform() or foreach()
methods, which give you an access to an RDD.
JavaDStream dataFrameStream =
ctx.textFileStream("source").transform(new Function2() {
@Override
public JavaRDD call(JavaRDD incomingRdd, Time
batchTime) throws Exception {
// Get an API for operating DataFrames
HiveContext ctx = new HiveContext(incomingRdd.context());
// create a schema for DataFrame (declare columns)
StructType schema = null;
// map incoming data into RDD of DataFrame's rows
JavaRDD rowsRdd = incomingRdd.map(rddMember -> new
GenericRow(100));
// DataFrame creation
DataFrame df = ctx.createDataFrame(rowsRdd, schema);
// here you may perform some operations on df, or return it as a stream
return df.toJavaRDD();
}
});
On Fri, Jun 3, 2016 at 5:44 PM, Zakaria Hili wrote:
> Hi,
> I m newbie in spark and I want to ask you a simple question.
> I have an JavaDStream which contains data selected from sql database.
> something like (id, user, score ...)
> and I want to convert the JavaDStream to a dataframe .
>
> how can I do this with java ?
> Thank you
> ᐧ
>
