Like this?
val rawXmls = ssc.fileStream(path, classOf[XmlInputFormat],
classOf[LongWritable],
classOf[Text])
Thanks
Best Regards
On Mon, Jun 22, 2015 at 5:45 PM, Yong Feng <fengyong...@gmail.com> wrote:
> Thanks a lot, Akhil
>
> I saw this mail thread before, but still do not understand how to use
> XmlInputFormatof mahout in Spark Streaming (I am not Spark Streaming
> Expert yet ;-)). Can you show me some sample code for explanation.
>
> Thanks in advance,
>
> Yong
>
> On Mon, Jun 22, 2015 at 6:44 AM, Akhil Das <ak...@sigmoidanalytics.com>
> wrote:
>
>> You can use fileStream for that, look at the XMLInputFormat
>> <https://github.com/apache/mahout/blob/ad84344e4055b1e6adff5779339a33fa29e1265d/examples/src/main/java/org/apache/mahout/classifier/bayes/XmlInputFormat.java>
>> of mahout. It should give you full XML object as on record, (as opposed to
>> an XML record spread across multiple line records in textFileStream). Also
>> this
>> thread
>> <http://apache-spark-user-list.1001560.n3.nabble.com/Parsing-a-large-XML-file-using-Spark-td19239.html>
>> has some discussion around it.
>>
>> Thanks
>> Best Regards
>>
>> On Mon, Jun 22, 2015 at 12:23 AM, Yong Feng <fengyong...@gmail.com>
>> wrote:
>>
>>>
>>> Hi Spark Experts
>>>
>>> I have a customer who wants to monitor coming data files (with xml
>>> format), and then analysize them after that put analysized data into DB.
>>> The size of each file is about 30MB (or even less in future). Spark
>>> streaming seems promising.
>>>
>>> After learning Spark Streaming and also google-ing how Spark Streaming
>>> handle xml files, I found there seems no existing Spark Stream utility to
>>> recognize whole xml file and parse it. The fileStream seems line-oriented.
>>> There is suggestion of putting whole xml file into one line, however it
>>> requires pre-processing files which will bring unexpected I/O.
>>>
>>> Can anyone throw some light on it? If will be great if there are some
>>> sample codes for me to start with.
>>>
>>> Thanks
>>>
>>> Yong
>>>
>>>
>>
>
