See also SPARK-3276 and SPARK-3553. Can you say more about the problem? what are the file timestamps, what happens when you run, what log messages if any are relevant. I do not expect there was any intended behavior change.
On Wed, Jan 21, 2015 at 5:17 AM, Terry Hole <[email protected]> wrote: > Hi, > > I am trying to move from 1.1 to 1.2 and found that the newFilesOnly=false > (Intend to include old files) does not work anymore. It works great in 1.1, > this should be introduced by the last change of this class. > > > > Does this flag behavior change or is it a regression? > > Issue should be caused by this code: > From line 157 in FileInputDStream.scala > val modTimeIgnoreThreshold = math.max( > initialModTimeIgnoreThreshold, // initial threshold based on > newFilesOnly setting > currentTime - durationToRemember.milliseconds // trailing end of > the remember window > ) > > > Regards > > - Terry > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
