Hi Arjun, yes, it is possible to start the web dashboard also when running jobs from the IDE. It is a bit hacky though...
You can do it by creating a LocalStreamExecutionEnvironment as follows: val config = new Configuration() config.setBoolean(ConfigConstants.LOCAL_START_WEBSERVER, true) config.setString(ConfigConstants.JOB_MANAGER_WEB_LOG_PATH_KEY, "/path/to/some/random/text/file") new LocalStreamEnvironment(config) The dashboard will try to access the job manager log file and terminate if it cannot be found. Since the JM log file is not present when running from the IDE, we need to point it to some other (text) file. I hope this will be fixed soon. Best, Fabian 2016-01-12 2:27 GMT+01:00 Arjun Rao <sporty.ar...@gmail.com>: > Thanks for the replies and help. Stephan, the Maven shortcut worked like a > charm :). > > - As for the 50ms window duration, when I was running the WindowWordCount > example with a duration of 5ms, I encountered this error stack trace: > > Exception in thread "main" java.lang.IllegalArgumentException: Window > length must be at least 50 msecs > at > > org.apache.flink.streaming.runtime.operators.windowing.AbstractAlignedProcessingTimeWindowOperator.<init>(AbstractAlignedProcessingTimeWindowOperator.java:84) > at > > org.apache.flink.streaming.runtime.operators.windowing.AggregatingProcessingTimeWindowOperator.<init>(AggregatingProcessingTimeWindowOperator.java:40) > at > > org.apache.flink.streaming.api.datastream.WindowedStream.createFastTimeOperatorIfValid(WindowedStream.java:616) > at > > org.apache.flink.streaming.api.datastream.WindowedStream.reduce(WindowedStream.java:146) > at > > org.apache.flink.streaming.api.datastream.WindowedStream.aggregate(WindowedStream.java:556) > at > > org.apache.flink.streaming.api.datastream.WindowedStream.sum(WindowedStream.java:373) > > - As for logging separate datastreams individually, the reason I ask is for > debugging purposes. Let us say that I have 5 different streaming jobs and 1 > taskmanager. If run out of the box, all these 5 jobs will log to the same > taskmanager log. My question is, is there a way to configure the logging > framework( or some other component perhaps) in such a way, that we are able > to log the different jobs separately to their own files, so it can be > easier to debug? > > - With respect to web page display in embedded mode, I apologize for the > confusion in terminology. When I say embedded, I mean running the flink > jobs through my IDE, with the flink jars in my local classpath. Is there a > way to look at the web page in this mode? Without having to deploy the > application jar to the flink cluster. Especially for the DataStream mode. > > Thanks! > > Best, > Arjun > > On Thu, Jan 7, 2016 at 10:37 AM, Stephan Ewen <se...@apache.org> wrote: > > > For the generated classes like > > "org.apache.flink.api.io.avro.generated.Address;", > > IntelliJ has a shortcut: Just right-click on the project, select "Maven > -> > > generate sources and update folders". > > > > That should do the trick... > > > > On Thu, Jan 7, 2016 at 11:55 AM, Till Rohrmann <trohrm...@apache.org> > > wrote: > > > > > Hi Arjun, > > > > > > welcome to the Flink community :-) > > > > > > On Thu, Jan 7, 2016 at 5:40 AM, Arjun Rao <sporty.ar...@gmail.com> > > wrote: > > > > > > > Hi, > > > > > > > > I am new to Apache Flink and I really like the look of the API. I > have > > > been > > > > working with Storm for the past year and have questions about the > > > > DataStream API among others. > > > > > > > > 1. What are the interactions of the actor system in the flink > > ecosystem? > > > > Where can I find more information? > > > > > > > > > > Actors are used internally by the system for the communication between > > the > > > different distributed components (Client, JobManager and TaskManager). > > > Thus, they are used to submit jobs, send job updates and delivering the > > > execution result for example. However, the actual data transfer of your > > job > > > is done using Netty. > > > > > > > > > > 2. Is there a Flink web UI in embedded mode? ( Not on cluster ) > > > > > > > > > > What do you mean with embedded mode? When you start Flink locally, then > > you > > > will also have a web UI. > > > > > > > > > > 3. Why is there a poll time min of 50ms for the Datastream API? > > > > > > > > > > Where do you see the poll time of 50 ms in the DataStream API? In order > > to > > > make the data transfer more efficient, data records are grouped > together > > > and shipped together. The size of this group is defined by the network > > > buffer size. Whenever such a buffer is full, it will be sent. However, > in > > > order to avoid that records are never sent, there is also a buffer > > timeout > > > after which the elements of the buffer are sent even if the buffer is > not > > > yet full. > > > > > > > > > > 4. Is it possible to log each datastream to a separate file?( i.e. > not > > > all > > > > > > in the same task manager log ). > > > > > > > > > > What do you mean by logging different DataStreams to separate files? At > > the > > > moment, all the logging which happens on the TaskManager is written to > > the > > > log file of this task manager. > > > > > > > > > > 5. I checked out the master from git, but I am unable to build the > > > project > > > > in Intellij, due to compilation errors in flink-staging. The root > cause > > > is > > > > lack of "generated" files of avro such as > > > > > > > > import org.apache.flink.api.io.avro.generated.Address; > > > > import org.apache.flink.api.io.avro.generated.Colors; > > > > import org.apache.flink.api.io.avro.generated.Fixed16; > > > > import org.apache.flink.api.io.avro.generated.User; > > > > > > > > These files are not being found. I have the avro plugin installed for > > > > Intellij. What else do I need to do to make the project > build/compile? > > > > > > > > > > I think the first time, you have to execute `mvn clean package > > -DskipTests > > > -Dmaven.javadoc.skip=true` on the command line in order to generate the > > > missing files. > > > > > > > > > > > > > > Appreciate the help! > > > > > > > > Best, > > > > > > > > Arjun > > > > > > > > > > Cheers, > > > Till > > > > > >
