Re: ResourceManager not using correct akka URI in standalone cluster (?)

[AJ Heller]

More information:

>From the master node, I cannot `telnet localhost 6123` nor `telnet  6123` while the cluster is apparently running. Connection refused
immediately. `netstat -n | grep 6123` is empty. There's no server
listening. But the processes are running on all machines.

Does it matter that I don't have hadoop or HDFS installed? It is optional,
right? To be clear, this fails at startup, long before I'm able to run any
job.

On Amazon EC2, the machines know of their private IPs, but not their public
IPs. I've instructed the cluster to operate over the public network because
I couldn't get the private IP scenario working.

Running `./bin/start-local.sh` shows non-zero counts in the Flink
Dashboard. Cluster setups show zero-counts all around.

-aj

On Thu, Sep 15, 2016 at 12:41 PM, AJ Heller  wrote:

> I'm running a standalone cluster on Amazon EC2. Leader election is
> happening according to the logs, and the Flink Dashboard is up and running,
> accessible remotely. The issue I'm having is that the SocketWordCount
> example is not working, the local connection is being refused!
>
> In the Flink Dashboard, 0 task managers are being reported. And in the
> jobmanager logs, the last line indicates "leader session null". All other
> akka URIs in the log file begin "akka.tcp://flink@PUBLIC_IP/...", but the
> Resourse Manager URI indicated "akka://flink/...".
>
>
> jobmanager log:
> http://pastebin.com/VWJM8XvW
>
> client log:
> http://pastebin.com/ZrWsbcwa
>
> flink-conf.yaml:
> http://pastebin.com/xy2tz7WS
>
> master and slave files are populated with public ips as well.
>

ResourceManager not using correct akka URI in standalone cluster (?)

[AJ Heller]

I'm running a standalone cluster on Amazon EC2. Leader election is
happening according to the logs, and the Flink Dashboard is up and running,
accessible remotely. The issue I'm having is that the SocketWordCount
example is not working, the local connection is being refused!

In the Flink Dashboard, 0 task managers are being reported. And in the
jobmanager logs, the last line indicates "leader session null". All other
akka URIs in the log file begin "akka.tcp://flink@PUBLIC_IP/...", but the
Resourse Manager URI indicated "akka://flink/...".


jobmanager log:
http://pastebin.com/VWJM8XvW

client log:
http://pastebin.com/ZrWsbcwa

flink-conf.yaml:
http://pastebin.com/xy2tz7WS

master and slave files are populated with public ips as well.

Re: Why tuples are not ignored after watermark?

[Fabian Hueske]

No, this is not possible unless you use an external service such as a
database.
The assigners might run on different machines and Flink does not provide
utilities for r/w shared state.

Best, Fabian

2016-09-15 20:17 GMT+02:00 Saiph Kappa :

> And is it possible to share state across parallel instances with
> AssignerWithPunctuatedWatermarks?
>
> Thanks!
>
> On Wed, Sep 14, 2016 at 9:52 AM, Aljoscha Krettek 
> wrote:
>
>> Hi,
>> the problem might be that your timestamp/watermark assigner is run in
>> parallel and that only one parallel instance of those operators emits the
>> watermark because only one of those parallel instances sees the element
>> with _3 == 9000. For the watermark to advance at an operator it needs to
>> advance in all upstream operations.
>>
>> Cheers,
>> Aljoscha
>>
>> On Fri, 9 Sep 2016 at 18:29 Saiph Kappa  wrote:
>>
>>> Hi,
>>>
>>> I have a streaming (event time) application where I am receiving events
>>> with the same assigned timestamp. I receive 1 events in total on a
>>> window of 5 minutes, but I emit water mark when 9000 elements have been
>>> received. This watermark is 6 minutes after the assigned timestamps. My
>>> question is: why the function that is associated with the window reads
>>> 1 elements and not 9000? All elements that have a timestamp lower than
>>> the watermark should be ignored (1000), but it's not happening.
>>>
>>> Here is part of the code:
>>> «
>>> val env = StreamExecutionEnvironment.getExecutionEnvironment
>>> env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
>>> val rawStream = env.socketTextStream("localhost", 4321)
>>>
>>> val punctuatedAssigner = new AssignerWithPunctuatedWatermarks[(String,
>>> Int, Long)] {
>>>   val timestamp = System.currentTimeMillis();
>>>
>>>   override def extractTimestamp(element: (String, Int, Long),
>>> previousElementTimestamp: Long): Long =
>>> timestamp
>>>
>>>   override def checkAndGetNextWatermark(lastElement: (String, Int,
>>> Long), extractedTimestamp: Long): Watermark = {
>>> if(lastElement._3 == 9000) {
>>>   val ts = extractedTimestamp + TimeUnit.MINUTES.toMillis(6)
>>>   new watermark.Watermark(ts)
>>> } else null
>>>   }
>>> }
>>>
>>> val stream = rawStream.map(line => {
>>>   val Array(p1, p2, p3) = line.split(" ")
>>>   (p1, p2.toInt, p3.toLong)
>>> })
>>>   .assignTimestampsAndWatermarks(punctuatedAssigner)
>>>
>>> stream.keyBy(1).timeWindow(Time.of(5, TimeUnit.MINUTES)).apply(function)
>>> »
>>>
>>> Thanks!
>>>
>>
>

Re: Data Transfer between TM should be encrypted

[Vijay Srinivasaraghavan]

 Hi Vinay,
There are some delays and we expect the PR to be created next week. 

RegardsVijay
On Wednesday, September 14, 2016 5:41 PM, vinay patil 
 wrote:
 

 Hi Vijay,
Did you raise the PR for this task, I don't mind testing it out as well.
Regards,Vinay Patil
On Tue, Aug 30, 2016 at 6:28 PM, Vinay Patil <[hidden email]> wrote:

Hi Vijay,
That's a good news for me. Eagerly waiting for this change so that I can 
integrate and test it before going live.
Regards,Vinay Patil
On Tue, Aug 30, 2016 at 4:06 PM, Vijay Srinivasaraghavan [via Apache Flink User 
Mailing List archive.] <[hidden email]> wrote:

 Hi Stephan,
The dev work is almost complete except the Yarn mode deployment stuff that 
needs to be patched. We are expecting to send a PR in a week or two.
RegardsVijay 

On Tuesday, August 30, 2016 12:39 AM, Stephan Ewen <[hidden email]> wrote:
 

 Let me loop in Vijay, I think he is the one working on this and can probably 
give the best estimate when it can be expected.
@vijay: For the SSL/TLS transport encryption - do you have an estimate for the 
timeline of that feature?

On Mon, Aug 29, 2016 at 8:54 PM, vinay patil <[hidden email]> wrote:

Hi Stephan,
Thank you for your reply.
Till when can I expect this feature to be integrated in master or release 
version ?

We are going to get production data (financial data) in October end , so want 
to have this feature before that.
Regards,Vinay Patil
On Mon, Aug 29, 2016 at 11:15 AM, Stephan Ewen [via Apache Flink User Mailing 
List archive.] <[hidden email]> wrote:

 Hi!
The way that the JIRA issue you linked will achieve this is by hooking into the 
network stream pipeline directly, and encrypt the raw network byte stream. We 
built the network stack on Netty, and will use Netty's SSL/TLS handlers for 
that.

That should be much more efficient than manual encryption/decryption in each 
user function.
Stephan





On Mon, Aug 29, 2016 at 6:12 PM, vinay patil <[hidden email]> wrote:

Hi Ufuk,
This is regarding this issuehttps://issues.apache.org/jira /browse/FLINK-4404

How can we achieve this, I am able to decrypt the data from Kafka coming in, 
but I want to make sure that the data is encrypted when flowing between TM's.
One approach I can think of is to decrypt the data at the start of each 
operator and encrypt it at the end of each operator, but I feel this is not an 
efficient approach.
I just want to check if there are alternatives to this and can this be achieved 
by doing some configurations.
Regards,Vinay Patil 
View this message in context: Data Transfer between TM should be encrypted
Sent from the Apache Flink User Mailing List archive. mailing list archive at 
Nabble.com.


 
 
   If you reply to this email, your message will be added to the discussion 
below: http://apache-flink-user-maili ng-list-archive.2336050.n4. 
nabble.com/Data-Transfer-betwe en-TM-should-be-encrypted- tp8781p8782.html   To 
start a new topic under Apache Flink User Mailing List archive., email [hidden 
email] 
 To unsubscribe from Apache Flink User Mailing List archive., click here.
 NAML 

 
View this message in context: Re: Data Transfer between TM should be encrypted
Sent from the Apache Flink User Mailing List archive. mailing list archive at 
Nabble.com.





 
   If you reply to this email, your message will be added to the discussion 
below: http://apache-flink-user-maili ng-list-archive.2336050.n4. 
nabble.com/Data-Transfer-betwe en-TM-should-be-encrypted- tp8781p8801.html   To 
start a new topic under Apache Flink User Mailing List archive., email [hidden 
email] 
 To unsubscribe from Apache Flink User Mailing List archive., click here.
 NAML 



 
View this message in context: Re: Data Transfer between TM should be encrypted
Sent from the Apache Flink User Mailing List archive. mailing list archive at 
Nabble.com.


   

Re: Streaming issue help needed

[Vaidyanathan Sivasubramanian]

Dear Marton,

Thanks very much for taking time to look into this issue.  Actually I am on 
latest Flink version (1.1.2).  Would the Hadoop version might be causing the 
problem (it is 2.6 with CDH 5.5.1)?  Would appreciate any more pointers to 
resolve this.

Thanks,
Vaidya.

> On 14-Sep-2016, at 4:53 PM, Márton Balassi  wrote:
> 
> Dear Vaidya,
> 
> This seems weird, me guess is that somehow that Time and AbstractTime 
> implementations are not from the same Flink version. 
> 
> According to your Maven build you should be using Flink 0.10.2. Since then 
> there have been changes to windowing, are you tied to that version or would 
> it be feasible to upgrade to the latest (1.1.2) just to have a bit more 
> information on the issue?
> 
> Best,
> 
> Marton
> 
> On Wed, Sep 14, 2016 at 8:51 AM, Vaidyanathan Sivasubramanian 
> > wrote:
> Hi,
> 
> I am trying to implement Flink steaming using an example in GitHub: 
> https://github.com/andi1400/bestXStreamRating 
> .  While executing I am 
> getting the below error.  Can you please help guide me on the resolution.  
> Thanks in advance!
> 
> 
> Starting execution of program
> 
> 
>  The program finished with the following exception:
> 
> java.lang.VerifyError: Bad type on operand stack
> Exception Details:
>   Location:
> 
> ca/uwaterloo/cs/bigdata2016w/andi1400/bestXStreamRating/AnalyzeTwitterBestXSentimentRatingFlink$.main([Ljava/lang/String;)V
>  @687: invokevirtual
>   Reason:
> Type 'org/apache/flink/streaming/api/windowing/time/Time' (current frame, 
> stack[2]) is not assignable to 
> 'org/apache/flink/streaming/api/windowing/time/AbstractTime'
>   Current Frame:
> bci: @687
> flags: { }
> locals: { 
> 'ca/uwaterloo/cs/bigdata2016w/andi1400/bestXStreamRating/AnalyzeTwitterBestXSentimentRatingFlink$',
>  '[Ljava/lang/String;', 
> 'ca/uwaterloo/cs/bigdata2016w/andi1400/bestXStreamRating/Conf', 
> 'java/lang/String', integer, 
> 'org/apache/flink/streaming/api/scala/StreamExecutionEnvironment', 
> 'org/apache/flink/api/scala/ExecutionEnvironment', 
> 'org/apache/flink/api/scala/DataSet', 'org/apache/flink/api/scala/DataSet', 
> 'ca/uwaterloo/cs/bigdata2016w/andi1400/bestXStreamRating/TermConfigurationFileScala',
>  '[Ljava/lang/String;', 'org/apache/flink/streaming/api/scala/DataStream', 
> 'scala/collection/immutable/Map', 
> 'org/apache/flink/streaming/api/scala/DataStream', integer, 
> 'org/apache/flink/streaming/api/scala/KeyedStream' }
> stack: { 'org/apache/flink/streaming/api/scala/KeyedStream', 
> 'org/apache/flink/streaming/api/windowing/time/Time', 
> 'org/apache/flink/streaming/api/windowing/time/Time' }
>   Bytecode:
> 000: bb00 4759 b200 4c2b c000 4eb6 0052 b700
> 010: 554d 2ab6 0057 bb00 5959 b700 5a12 5cb6
> 020: 0060 2cb6 0064 b600 6ab6 0060 b600 6eb6
> 030: 0074 2ab6 0057 bb00 5959 b700 5a12 76b6
> 040: 0060 2cb6 0079 b600 6ab6 0060 b600 6eb6
> 050: 0074 2ab6 0057 bb00 5959 b700 5a12 7bb6
> 060: 0060 2cb6 007e b600 6ab6 0060 b600 6eb6
> 070: 0074 2ab6 0057 bb00 5959 b700 5a12 80b6
> 080: 0060 2cb6 0083 b600 6ab6 0060 b600 6eb6
> 090: 0074 1285 b800 89b2 008f b600 9312 95b8
> 0a0: 0089 b200 8fb6 0093 2cb6 0098 b600 6ac0
> 0b0: 009a 4e2c b600 9db6 006a b800 a399 0007
> 0c0: 03a7 0004 0436 04b2 00a8 b600 ac3a 05b2
> 0d0: 00b1 b600 b43a 0619 062c b600 7eb6 006a
> 0e0: c000 9a19 06b6 00b9 b600 bd3a 0719 07bb
> 0f0: 00bf 59b7 00c0 bb00 c259 b700 c3b2 00c8
> 100: 12ca b600 cdb6 00d3 3a08 bb00 d559 1908
> 110: b600 d9b2 004c b600 ddb9 00e3 0200 b700
> 120: e63a 0919 09b6 00ea 3a0a 2cb6 00ed b600
> 130: 6ab8 00a3 9900 2119 05bb 00ef 5919 0ab7
> 140: 00f1 b200 c812 f3b6 00cd 12f3 b800 f9b6
> 150: 00ff a700 5d19 05bb 0101 5919 0a2c b601
> 160: 04b6 006a c000 9a2c b601 07b6 006a c000
> 170: 9a2c b601 0ab6 006a c000 9a2c b601 0db6
> 180: 006a c000 9a2c b600 98b6 006a c000 9a2c
> 190: b601 10b6 006a b801 1413 0116 b701 19b2
> 1a0: 00c8 12f3 b600 cd12 f3b8 00f9 b600 ff3a
> 1b0: 0b19 062c b601 1cb6 006a c000 9a19 06b6
> 1c0: 00b9 b600 bdbb 011e 59b7 011f bb01 2159
> 1d0: b701 22b2 00c8 12ca b600 cdb6 0125 b600
> 1e0: d9bb 0127 59b7 0128 b901 2c02 00bb 012e
> 1f0: 59b7 012f b901 3402 00bb 0136 59b7 0137
> 200: b201 3cb6 0140 b901 4303 00c0 0131 3a0c
> 210: 190b bb01 4559 b701 4612 9ab8 014c b200
> 220: c812 9ab6 00cd b601 51bb 0153 59b7 0154
> 230: 129a b801 4cb2 00c8 129a b600 cdb6 0151
> 240: bb01 5659 b701 57b6 015b 3a0d 2cb6 0083
> 250: b600 6ab8 00a3 360e 190d bb01 5d59 1909
> 260: 190c 150e b701 60bb 0162 59b7 0163 b200
> 270: c812