Hive bug? about no such table

2015-12-17 Thread Philip Lee
I think It is from Hive Bug about something related to metastore. Here is the thing. After I generated scale factor 300 named bigbench300 and bigbench100, which already existed before, I run "hive job with bigbench300". At first it was really fine. Then I run hive job with bigbench100 again. It w

Re: Specify jobmanager port in HA mode

2015-12-17 Thread Cory Monty
Hey Ufuk, We can try building it, but it might be a week or so given the holidays. We're currently targeting development against 1.0.0-SNAPSHOT and that release is OK unless 1.0 release is more than 2-3 months away. Let me know if you still want us to build your PR and test it out. Cheers, Cor

Re: Problem to show logs in task managers

2015-12-17 Thread Till Rohrmann
Hi Ana, you can simply modify the `log4j.properties` file in the `conf` directory. It should be automatically included in the Yarn application. Concerning your logging problem, it might be that you have set the logging level too high. Could you share the code with us? Cheers, Till On Thu, Dec 1

Re: global watermark across multiple kafka consumers

2015-12-17 Thread Stephan Ewen
@Andrew: Just to make sure that there is no confusion: Even though every Kafka Source generates a local watermark, these watermarks are merged when the streams are merged, for example in union() or keyBy() steps. The operator that merges streams tracks the current watermark of each stream and then

Re: All datanodes are bad

2015-12-17 Thread Stephan Ewen
I guess the problem is pretty much like exception message says: HDFS output does not work because all datanodes are bad. If you activate execution retries, the system would re-execute the job. If the datanodes would be well then, the job would succeed. Greetings, Stephan On Wed, Dec 16, 2015 at

Problem to show logs in task managers

2015-12-17 Thread Ana M. Martinez
Hi flink community, I am trying to show log messages using log4j. It works fine overall except for the messages I want to show in an inner class that implements org.apache.flink.api.common.aggregators.ConvergenceCriterion. I am very new to this, but it seems that I’m having problems to show the

Usecase for Flink

2015-12-17 Thread igor.berman
Hi, We are looking at Flink and trying to understand if our usecase is relevant to it. We need process stream of events. Each event is for some id(e.g. device id), when each event should be 1. stored in some persistent storage(e.g. cassandra) 2. previously persisted events should be fetched and s