Entire file in a window.
On Mon, Nov 10, 2014 at 9:20 AM, Saiph Kappa <saiph.ka...@gmail.com> wrote:
> Hi,
>
> In my application I am doing something like this "new
> StreamingContext(sparkConf, Seconds(10)).textFileStream("logs/")", and I
> get some unknown exceptions when I copy a file with about 800 MB to that
> folder ("logs/"). I have a single worker running with 512 MB of memory.
>
> Anyone can tell me if every 10 seconds spark reads parts of that big file,
> or if it attempts to read the entire file in a single window? How does it
> work?
>
> Thanks.
>
>
