[jira] [Commented] (FLINK-3987) Add data stream connector to DistributedLog

2016-09-20 Thread John Lonergan (JIRA)

[ 
https://issues.apache.org/jira/browse/FLINK-3987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507255#comment-15507255
 ] 

John Lonergan commented on FLINK-3987:
--

Anyone fancy collaborating?

> Add data stream connector to DistributedLog
> ---
>
> Key: FLINK-3987
> URL: https://issues.apache.org/jira/browse/FLINK-3987
> Project: Flink
>  Issue Type: Improvement
>Reporter: Jia Zhai
>
> I would like to add a connector to DistributedLog, which recently published 
> by twitter.
> All the infomation of DistributedLog, could be found at here: 
> http://distributedlog.io/html
> And this JIRA ticket is to track the data stream connector



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (FLINK-3988) Add data set connector to DistributedLog

2016-09-20 Thread John Lonergan (JIRA)

[ 
https://issues.apache.org/jira/browse/FLINK-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507259#comment-15507259
 ] 

John Lonergan commented on FLINK-3988:
--

Anyone fancy collaborating?

> Add data set connector to DistributedLog
> 
>
> Key: FLINK-3988
> URL: https://issues.apache.org/jira/browse/FLINK-3988
> Project: Flink
>  Issue Type: Improvement
>Reporter: Jia Zhai
>
> I would like to add a connector to DistributedLog, which recently published 
> by twitter.
> All the infomation of DistributedLog, could be found at here: 
> http://distributedlog.io/html
> And this JIRA ticket is to track the data set connector



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (FLINK-13588) StreamTask.handleAsyncException throws away the exception cause

2019-08-15 Thread John Lonergan (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-13588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908529#comment-16908529
 ] 

John Lonergan commented on FLINK-13588:
---

See [https://github.com/apache/flink/pull/9456]

Hi done the work including test - trivial change.

Unfortunately I cannot verify the test as I couldn't work out how to make the 
existing build including tests on master run to completion without tests 
hanging for ages, and loads of errors.

I am using Java 8 221

Tried maven 3.1.1 and 3.2.5

No idea how to fix.

The following works but doesn't run tests 

{{mvn clean package -DskipTests # this will take up to 10 minutes}}



Also couldn't run test in IntelliJ getting error 

Error:java: invalid flag: --add-exports=java.base/sun.net.util=ALL-UNNAMED

> StreamTask.handleAsyncException throws away the exception cause
> ---
>
> Key: FLINK-13588
> URL: https://issues.apache.org/jira/browse/FLINK-13588
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / Task
>Affects Versions: 1.8.1
>Reporter: John Lonergan
>Assignee: John Lonergan
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Code below throws the reason 'message' away making it hard to diagnose why a 
> split has failed for instance.
>  
> {code:java}
> https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L909
> @Override
>   public void handleAsyncException(String message, Throwable exception) {
>   if (isRunning) {
>   // only fail if the task is still running
>   getEnvironment().failExternally(exception);
>   }
> }{code}
>  
> Need to pass the message through so that we see it in logs please.
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (FLINK-13588) StreamTask.handleAsyncException throws away the exception cause

2019-08-14 Thread John Lonergan (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-13588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907109#comment-16907109
 ] 

John Lonergan commented on FLINK-13588:
---

Hi yes that's how we"fixed" it because it limits the change to a one liner.





> StreamTask.handleAsyncException throws away the exception cause
> ---
>
> Key: FLINK-13588
> URL: https://issues.apache.org/jira/browse/FLINK-13588
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / Task
>Affects Versions: 1.8.1
>Reporter: John Lonergan
>Priority: Major
>
> Code below throws the reason 'message' away making it hard to diagnose why a 
> split has failed for instance.
>  
> {code:java}
> https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L909
> @Override
>   public void handleAsyncException(String message, Throwable exception) {
>   if (isRunning) {
>   // only fail if the task is still running
>   getEnvironment().failExternally(exception);
>   }
> }{code}
>  
> Need to pass the message through so that we see it in logs please.
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Comment Edited] (FLINK-10382) Writer has already been opened while using AvroKeyValueSinkWriter and BucketingSink

2019-08-19 Thread John Lonergan (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-10382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910497#comment-16910497
 ] 

John Lonergan edited comment on FLINK-10382 at 8/19/19 3:48 PM:


Just fell over this same problem - Fundamentally this class doesn't work as far 
as I can tell.

Presumably there are missing tests - which raises a question around QA in Flink 
- what is the overall quality of testing in Flink?

I suggest that rather than just documenting this class as deprecated it comes 
with a comment saying it's broken and should be avoided - or just delete it 
entirely if this class has been abandoned.

---

As far as a "fix" I think something like this ought to work. Flush needed first 
to avoid potential data loss.
 

@Override public void close() throws IOException { 

   if (keyValueWriter != null) {

     flush(); 

     super.close();

     keyValueWriter = null;

  } else { // need to make sure we close this if we never created the Key/Value 
Writer.

      super.close();

  }

}


Is the bucketing sink abandoned?

This close issue only seems to be safe on StringWriter and SequenceFileWriter, 
the first because there's nothing to close and the second because it knows 
whether it owns the stream or not.




was (Author: johnlon):
Just fell over this same problem - Fundamentally this class doesn't work as far 
as I can tell.

Presumably there are missing tests - which raises a question around QA in Flink 
- what is the overall quality of testing in Flink?

I suggest that rather than just documenting this class as deprecated it comes 
with a comment saying it's broken and should be avoided - or just delete it 
entirely if this class has been abandoned.

---

As far as a "fix" I think something like this ought to work. Flush needed first 
to avoid potential data loss.
 
{{
@Override public void close() throws IOException { 
   if (keyValueWriter != null) {
     flush(); 
     super.close();
     keyValueWriter = null;
  } else { // need to make sure we close this if we never created the Key/Value 
Writer.
      super.close();
  }
}
}}


Is the bucketing sink abandoned?

This close issue only seems to be safe on StringWriter and SequenceFileWriter, 
the first because there's nothing to close and the second because it knows 
whether it owns the stream or not.



> Writer has already been opened while using AvroKeyValueSinkWriter and 
> BucketingSink
> ---
>
> Key: FLINK-10382
> URL: https://issues.apache.org/jira/browse/FLINK-10382
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / FileSystem
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Chengzhi Zhao
>Priority: Major
>
> I am using flink 1.6.0 and I am using AvroKeyValueSinkWriter and 
> BucketingSink to S3.
>  
> After the application running for a while ~ 20 mins, I got an *exception: 
> java.lang.IllegalStateException: Writer has already been opened*
> {code:java}
> 2018-09-17 15:40:23,771 INFO 
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering 
> checkpoint 4 @ 1537198823640 for job 8f9ab122fb7452714465eb1e1989e4d7.
> 2018-09-17 15:41:27,805 INFO 
> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Unnamed (2/16) 
> (25914cb3f77c8e4271b0fb6ea597ed50) switched from RUNNING to FAILED.
> java.lang.IllegalStateException: Writer has already been opened
> at 
> org.apache.flink.streaming.connectors.fs.StreamWriterBase.open(StreamWriterBase.java:68)
> at 
> org.apache.flink.streaming.connectors.fs.AvroKeyValueSinkWriter.open(AvroKeyValueSinkWriter.java:150)
> at 
> org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.openNewPartFile(BucketingSink.java:583)
> at 
> org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.invoke(BucketingSink.java:458)
> at 
> org.apache.flink.streaming.api.functions.sink.SinkFunction.invoke(SinkFunction.java:52)
> at 
> org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:56)
> at 
> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:202)
> at 
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:104)
> at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:306)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:703)
> at java.lang.Thread.run(Thread.java:748)
> 2018-09-17 15:41:27,808 INFO 
> org.apache.flink.runtime.executiongraph.ExecutionGraph - Job Stream to Stream 
> Join (8f9ab122fb7452714465eb1e1989e4d7) switched from state RUNNING to 
> FAILING.
> java.lang.IllegalStateException: Writer has already been opened
> at 
> org.apache.flink.streaming.connectors.fs.StreamWriterBase.open(StreamWriterBase.java:68)
> at 
> 

[jira] [Comment Edited] (FLINK-10382) Writer has already been opened while using AvroKeyValueSinkWriter and BucketingSink

2019-08-19 Thread John Lonergan (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-10382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910497#comment-16910497
 ] 

John Lonergan edited comment on FLINK-10382 at 8/19/19 3:47 PM:


Just fell over this same problem - Fundamentally this class doesn't work as far 
as I can tell.

Presumably there are missing tests - which raises a question around QA in Flink 
- what is the overall quality of testing in Flink?

I suggest that rather than just documenting this class as deprecated it comes 
with a comment saying it's broken and should be avoided - or just delete it 
entirely if this class has been abandoned.

---

As far as a "fix" I think something like this ought to work. Flush needed first 
to avoid potential data loss.
 
{{
@Override public void close() throws IOException { 
   if (keyValueWriter != null) {
     flush(); 
     super.close();
     keyValueWriter = null;
  } else { // need to make sure we close this if we never created the Key/Value 
Writer.
      super.close();
  }
}
}}


Is the bucketing sink abandoned?

This close issue only seems to be safe on StringWriter and SequenceFileWriter, 
the first because there's nothing to close and the second because it knows 
whether it owns the stream or not.




was (Author: johnlon):
Just fell over this same problem - Fundamentally this class doesn't work as far 
as I can tell.

Presumably there are missing tests - which raises a question around QA in Flink 
- what is the overall quality of testing in Flink?

I suggest that rather than just documenting this class as deprecated it comes 
with a comment saying it's broken and should be avoided - or just delete it 
entirely if this class has been abandoned.

---

As far as a "fix" I think something like this ought to work. Flush needed first 
to avoid potential data loss.
 
{{@Override public void close() throws IOException { 
   if (keyValueWriter != null) {
     flush(); 
     super.close();
     keyValueWriter = null;
  } else { // need to make sure we close this if we never created the Key/Value 
Writer.
      super.close();
  }
}
}}


Is the bucketing sink abandoned?

This close issue only seems to be safe on StringWriter and SequenceFileWriter, 
the first because there's nothing to close and the second because it knows 
whether it owns the stream or not.



> Writer has already been opened while using AvroKeyValueSinkWriter and 
> BucketingSink
> ---
>
> Key: FLINK-10382
> URL: https://issues.apache.org/jira/browse/FLINK-10382
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / FileSystem
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Chengzhi Zhao
>Priority: Major
>
> I am using flink 1.6.0 and I am using AvroKeyValueSinkWriter and 
> BucketingSink to S3.
>  
> After the application running for a while ~ 20 mins, I got an *exception: 
> java.lang.IllegalStateException: Writer has already been opened*
> {code:java}
> 2018-09-17 15:40:23,771 INFO 
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering 
> checkpoint 4 @ 1537198823640 for job 8f9ab122fb7452714465eb1e1989e4d7.
> 2018-09-17 15:41:27,805 INFO 
> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Unnamed (2/16) 
> (25914cb3f77c8e4271b0fb6ea597ed50) switched from RUNNING to FAILED.
> java.lang.IllegalStateException: Writer has already been opened
> at 
> org.apache.flink.streaming.connectors.fs.StreamWriterBase.open(StreamWriterBase.java:68)
> at 
> org.apache.flink.streaming.connectors.fs.AvroKeyValueSinkWriter.open(AvroKeyValueSinkWriter.java:150)
> at 
> org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.openNewPartFile(BucketingSink.java:583)
> at 
> org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.invoke(BucketingSink.java:458)
> at 
> org.apache.flink.streaming.api.functions.sink.SinkFunction.invoke(SinkFunction.java:52)
> at 
> org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:56)
> at 
> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:202)
> at 
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:104)
> at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:306)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:703)
> at java.lang.Thread.run(Thread.java:748)
> 2018-09-17 15:41:27,808 INFO 
> org.apache.flink.runtime.executiongraph.ExecutionGraph - Job Stream to Stream 
> Join (8f9ab122fb7452714465eb1e1989e4d7) switched from state RUNNING to 
> FAILING.
> java.lang.IllegalStateException: Writer has already been opened
> at 
> org.apache.flink.streaming.connectors.fs.StreamWriterBase.open(StreamWriterBase.java:68)
> at 
> 

[jira] [Commented] (FLINK-10382) Writer has already been opened while using AvroKeyValueSinkWriter and BucketingSink

2019-08-19 Thread John Lonergan (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-10382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910497#comment-16910497
 ] 

John Lonergan commented on FLINK-10382:
---

Just fell over this same problem - Fundamentally this class doesn't work as far 
as I can tell.

Presumably there are missing tests - which raises a question around QA in Flink 
- what is the overall quality of testing in Flink?

I suggest that rather than just documenting this class as deprecated it comes 
with a comment saying it's broken and should be avoided - or just delete it 
entirely if this class has been abandoned.

---

As far as a "fix" I think something like this ought to work. Flush needed first 
to avoid potential data loss.
 
{{@Override public void close() throws IOException { 
   if (keyValueWriter != null) {
     flush(); 
     super.close();
     keyValueWriter = null;
  } else { // need to make sure we close this if we never created the Key/Value 
Writer.
      super.close();
  }
}
}}


Is the bucketing sink abandoned?

This close issue only seems to be safe on StringWriter and SequenceFileWriter, 
the first because there's nothing to close and the second because it knows 
whether it owns the stream or not.



> Writer has already been opened while using AvroKeyValueSinkWriter and 
> BucketingSink
> ---
>
> Key: FLINK-10382
> URL: https://issues.apache.org/jira/browse/FLINK-10382
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / FileSystem
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Chengzhi Zhao
>Priority: Major
>
> I am using flink 1.6.0 and I am using AvroKeyValueSinkWriter and 
> BucketingSink to S3.
>  
> After the application running for a while ~ 20 mins, I got an *exception: 
> java.lang.IllegalStateException: Writer has already been opened*
> {code:java}
> 2018-09-17 15:40:23,771 INFO 
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering 
> checkpoint 4 @ 1537198823640 for job 8f9ab122fb7452714465eb1e1989e4d7.
> 2018-09-17 15:41:27,805 INFO 
> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Unnamed (2/16) 
> (25914cb3f77c8e4271b0fb6ea597ed50) switched from RUNNING to FAILED.
> java.lang.IllegalStateException: Writer has already been opened
> at 
> org.apache.flink.streaming.connectors.fs.StreamWriterBase.open(StreamWriterBase.java:68)
> at 
> org.apache.flink.streaming.connectors.fs.AvroKeyValueSinkWriter.open(AvroKeyValueSinkWriter.java:150)
> at 
> org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.openNewPartFile(BucketingSink.java:583)
> at 
> org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.invoke(BucketingSink.java:458)
> at 
> org.apache.flink.streaming.api.functions.sink.SinkFunction.invoke(SinkFunction.java:52)
> at 
> org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:56)
> at 
> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:202)
> at 
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:104)
> at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:306)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:703)
> at java.lang.Thread.run(Thread.java:748)
> 2018-09-17 15:41:27,808 INFO 
> org.apache.flink.runtime.executiongraph.ExecutionGraph - Job Stream to Stream 
> Join (8f9ab122fb7452714465eb1e1989e4d7) switched from state RUNNING to 
> FAILING.
> java.lang.IllegalStateException: Writer has already been opened
> at 
> org.apache.flink.streaming.connectors.fs.StreamWriterBase.open(StreamWriterBase.java:68)
> at 
> org.apache.flink.streaming.connectors.fs.AvroKeyValueSinkWriter.open(AvroKeyValueSinkWriter.java:150)
> at 
> org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.openNewPartFile(BucketingSink.java:583)
> at 
> org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.invoke(BucketingSink.java:458)
> at 
> org.apache.flink.streaming.api.functions.sink.SinkFunction.invoke(SinkFunction.java:52)
> at 
> org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:56)
> at 
> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:202)
> at 
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:104)
> at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:306)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:703)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> After checking the code, I think the issue might be related to 
> AvroKeyValueSinkWriter.java and led to the writer has not been closed 
> completely. I also noticed this change and affect 1.5+ 
> 

[jira] [Created] (FLINK-14121) upgrade tocommons-compress:1.19 due to CVE

2019-09-18 Thread John Lonergan (Jira)
John Lonergan created FLINK-14121:
-

 Summary: upgrade tocommons-compress:1.19 due to CVE
 Key: FLINK-14121
 URL: https://issues.apache.org/jira/browse/FLINK-14121
 Project: Flink
  Issue Type: Improvement
  Components: Build System, Release System
Affects Versions: 1.9.0
Reporter: John Lonergan


See 
https://commons.apache.org/proper/commons-compress/security-reports.html




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (FLINK-14088) Couchbase connector - sink with checkpointing support

2019-09-16 Thread John Lonergan (Jira)
John Lonergan created FLINK-14088:
-

 Summary: Couchbase connector - sink with checkpointing support
 Key: FLINK-14088
 URL: https://issues.apache.org/jira/browse/FLINK-14088
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Common
Reporter: John Lonergan


https://flink.apache.org/ecosystem.html - Seeing no reference to Couchbase

Is this planned?

Looking for a sink with checkpointing support.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Created] (FLINK-13588) StreamTask.handleAsyncException throws away the exception cause

2019-08-05 Thread John Lonergan (JIRA)
John Lonergan created FLINK-13588:
-

 Summary: StreamTask.handleAsyncException throws away the exception 
cause
 Key: FLINK-13588
 URL: https://issues.apache.org/jira/browse/FLINK-13588
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.8.1
Reporter: John Lonergan


Code below throws the reason 'message' away making it hard to diagnose why a 
split has failed for instance.

 
{code:java}

https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L909

@Override
public void handleAsyncException(String message, Throwable exception) {
if (isRunning) {
// only fail if the task is still running
getEnvironment().failExternally(exception);
}
}{code}
 

Need to pass the message through so that we see it in logs please.

 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (FLINK-13588) StreamTask.handleAsyncException throws away the exception cause

2019-08-08 Thread John Lonergan (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-13588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903278#comment-16903278
 ] 

John Lonergan commented on FLINK-13588:
---

Don't agree.

If there is context then the code should not throw it away. Principal.

Without the exception message I cannot discover why the split failed.

For example we had a failure because of a zero byte avro file in hdfs. The
error message had the filename in it but the code throws it away.

As a result we had to write and run a separate trivial job that brute
forced reading all the files (100k) without flinks help.

The change is justified.
I don't think it's reasonable to throw this away. It looks like the error
handling /logging is a bit inconsistent for sure.

We are now running with a modified version of this class that wraps the
original exception into a runtime exception that includes the cause text.









> StreamTask.handleAsyncException throws away the exception cause
> ---
>
> Key: FLINK-13588
> URL: https://issues.apache.org/jira/browse/FLINK-13588
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 1.8.1
>Reporter: John Lonergan
>Priority: Major
>
> Code below throws the reason 'message' away making it hard to diagnose why a 
> split has failed for instance.
>  
> {code:java}
> https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L909
> @Override
>   public void handleAsyncException(String message, Throwable exception) {
>   if (isRunning) {
>   // only fail if the task is still running
>   getEnvironment().failExternally(exception);
>   }
> }{code}
>  
> Need to pass the message through so that we see it in logs please.
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (FLINK-14238) ParquetPojoInputFormat logging bug

2019-09-26 Thread John Lonergan (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-14238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Lonergan updated FLINK-14238:
--
Issue Type: Bug  (was: Improvement)

> ParquetPojoInputFormat logging bug
> --
>
> Key: FLINK-14238
> URL: https://issues.apache.org/jira/browse/FLINK-14238
> Project: Flink
>  Issue Type: Bug
>  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.9.0
>Reporter: John Lonergan
>Priority: Minor
>
> Line 64: LOG.error("Fields number is %d", getFieldNames().length);
> Generates a bunch of spurious error logging.
> It is a coding error
> - should probably have been trace but ...
> - is actually a coding error as "%d" isn't a valid format for the logger so 
> we never see the length
> Recommend - This line should be deleted.
> https://github.com/apache/flink/blob/d015ce7a3b86a529f4db79ed8ac8dbe28c62d6b8/flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/ParquetPojoInputFormat.java#L64



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (FLINK-14238) ParquetPojoInputFormat logging bug

2019-09-26 Thread John Lonergan (Jira)
John Lonergan created FLINK-14238:
-

 Summary: ParquetPojoInputFormat logging bug
 Key: FLINK-14238
 URL: https://issues.apache.org/jira/browse/FLINK-14238
 Project: Flink
  Issue Type: Improvement
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.9.0
Reporter: John Lonergan


Line 64: LOG.error("Fields number is %d", getFieldNames().length);

Generates a bunch of spurious error logging.

It is a coding error
- should probably have been trace but ...
- is actually a coding error as "%d" isn't a valid format for the logger so we 
never see the length

Recommend - This line should be deleted.

https://github.com/apache/flink/blob/d015ce7a3b86a529f4db79ed8ac8dbe28c62d6b8/flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/ParquetPojoInputFormat.java#L64



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (FLINK-14170) Support hadoop < 2.7 with StreamingFileSink.BulkFormatBuilder

2019-09-23 Thread John Lonergan (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-14170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935904#comment-16935904
 ] 

John Lonergan commented on FLINK-14170:
---

Yep it's unnecessarily restrictive and actually breaks Parquest even though it 
would otherwise work just fine on Hadoop 2.6.

Remove the global check in construction and instead the make the code throw an 
"NotImplementedException" +only+ if a sink actually happens to make that call.

> Support hadoop < 2.7 with StreamingFileSink.BulkFormatBuilder
> -
>
> Key: FLINK-14170
> URL: https://issues.apache.org/jira/browse/FLINK-14170
> Project: Flink
>  Issue Type: Improvement
>  Components: API / DataSet
>Affects Versions: 1.8.0, 1.8.1, 1.8.2, 1.9.0
>Reporter: Bhagavan
>Priority: Major
>
> Currently, StreamingFileSink is supported only with Hadoop >= 2.7 
> irrespective of Row/bulk format builder. This restriction is due to truncate 
> is not supported in  Hadoop < 2.7
> However, BulkFormatBuilder does not use truncate method to restore the file. 
> So the restricting StreamingFileSink.BulkFormatBuilder to be used only with 
> Hadoop >= 2.7 is not necessary.
> So requested improvement is to remove the precondition on 
> HadoopRecoverableWriter and allow  BulkFormatBuilder (Parquet) to be used in 
> Hadoop 2.6 ( Most of the enterprises still on CDH 5.x)
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (FLINK-14174) Don't swallow exception when rethrowing type mismatches with side outputs

2019-09-23 Thread John Lonergan (Jira)
John Lonergan created FLINK-14174:
-

 Summary: Don't swallow exception when rethrowing type mismatches 
with side outputs
 Key: FLINK-14174
 URL: https://issues.apache.org/jira/browse/FLINK-14174
 Project: Flink
  Issue Type: Improvement
  Components: API / DataStream
Affects Versions: 1.9.0, 1.8.1
Reporter: John Lonergan


The change made by https://github.com/apache/flink/pull/4663/files swallows the 
original exception.

Whilst I am in favour of adding additional helpful tips (which was the purpose 
of FLINK-4663) I don't agree with throwing away or masking causal exceptions.

IMHO the correct approach is to add the helpful hint as the first arg to "new 
ExceptionInChainedOperatorException(msg, ex)" and pass the original class cast 
ex as the cause.

Ie change this .. 
https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/OperatorChain.java#L672




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (FLINK-14174) Don't swallow exception when rethrowing type mismatches with side outputs

2019-09-23 Thread John Lonergan (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-14174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Lonergan updated FLINK-14174:
--
Issue Type: Bug  (was: Improvement)

> Don't swallow exception when rethrowing type mismatches with side outputs
> -
>
> Key: FLINK-14174
> URL: https://issues.apache.org/jira/browse/FLINK-14174
> Project: Flink
>  Issue Type: Bug
>  Components: API / DataStream
>Affects Versions: 1.8.1, 1.9.0
>Reporter: John Lonergan
>Priority: Major
>
> The change made by https://github.com/apache/flink/pull/4663/files introduces 
> a "helpful hint" that a class cast at that location might be due a Flink 
> pipeline cfg error and the code goes onto swallows the original exception and 
> masked a protocol buffer serialisation problem we were having.
> This "helpful hint" is sometimes a complete red herring and in my case wasted 
> a lot of peoples time.
> In my case I had a class cast error in some proto serialisation code and 
> because the "helpful hint" traps ClassCastException I wasn't able to discover 
> the error easily. In the end we modified the Flink distribution to remove 
> this "helpful hint" at which point the real error was found and we quickly 
> fixed it - but not without a lot of burned time.
> I am not convinced of the cost/benefit of the "helpful hint" introduced by 
> FLINK-4663 for two reasons 
> - it can be a red herring - in mine case and also and [at least one other 
> person 
> |https://stackoverflow.com/questions/56069797/multiple-outputtags-in-stream-process-function-with-classcastexception
> ]
> - I don't agree with ever throwing away or masking causal exceptions - these 
> must always be propagated (I raised a similar issue in my previous 
> contribution)
> 
> My suggestion is to either back out FLINK-4663 so that we get to see the raw 
> underlying exception and call stack -or- come up with a way to distinguish 
> the specific case "FLINK-4663" was attempting to cover and only emit that 
> hint hint if the specific case is encountered. 
> For all other cases the helpful hint should not be emitted.  
> And - regardless of whether the helpful hint it emitted or not +the causal 
> exception must always be propagated+.
> My vote is to back out FLINK-4663 and maybe add some logging instead.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (FLINK-14174) Don't swallow exception when rethrowing type mismatches with side outputs

2019-09-23 Thread John Lonergan (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-14174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Lonergan updated FLINK-14174:
--
Description: 
The change made by https://github.com/apache/flink/pull/4663/files introduces a 
"helpful hint" that a class cast at that location might be due a Flink pipeline 
cfg error and the code goes onto swallows the original exception and masked a 
protocol buffer serialisation problem we were having.

This "helpful hint" is sometimes a complete red herring and in my case wasted a 
lot of peoples time.

In my case I had a class cast error in some proto serialisation code and 
because the "helpful hint" traps ClassCastException I wasn't able to discover 
the error easily. In the end we modified the Flink distribution to remove this 
"helpful hint" at which point the real error was found and we quickly fixed it 
- but not without a lot of burned time.

I am not convinced of the cost/benefit of the "helpful hint" introduced by 
FLINK-4663 for two reasons 
- it can be a red herring - in mine case and also and [at least one other 
person 
|https://stackoverflow.com/questions/56069797/multiple-outputtags-in-stream-process-function-with-classcastexception
]
- I don't agree with ever throwing away or masking causal exceptions - these 
must always be propagated (I raised a similar issue in my previous contribution)



My suggestion is to either back out FLINK-4663 so that we get to see the raw 
underlying exception and call stack -or- come up with a way to distinguish the 
specific case "FLINK-4663" was attempting to cover and only emit that hint hint 
if the specific case is encountered. 
For all other cases the helpful hint should not be emitted.  
And - regardless of whether the helpful hint it emitted or not +the causal 
exception must always be propagated+.

My vote is to back out FLINK-4663 and maybe add some logging instead.


  was:
The change made by https://github.com/apache/flink/pull/4663/files introduces a 
"helpful hint" that a class cast at that location might be due a Flink pipeline 
cfg error and the code goes onto swallows the original exception and masked a 
protocol buffer serialisation problem we were having.

This "helpful hint" is sometimes a complete red herring and in my case wasted a 
lot of peoples time.

In my case I had a class cast error in some proto serialisation code and 
because the "helpful hint" traps ClassCastException I wasn't able to discover 
the error easily. In the end we modified the Flink distribution to remove this 
"helpful hint" at which point the real error was found and we quickly fixed it 
- but not without a lot of burned time.

I am not convinced of the cost/benefit of the "helpful hint" introduced by 
FLINK-4663 for two reasons 
- it can be a red herring - in mine case and also and [at least one other 
person 
|https://stackoverflow.com/questions/56069797/multiple-outputtags-in-stream-process-function-with-classcastexception
]
- I don't agree with ever throwing away or masking causal exceptions - these 
must always be propagated (I raised a similar issue in my previous contribution)



My suggestion is to either back out FLINK-4663 so that we get to see the raw 
underlying exception and call stack or come up with a way to distinguish the 
specific case "FLINK-4663" was attempting to cover and only emit that hint hint 
if the specific case is encountered. 
For all other cases the helpful hint should not be emitted.  
And - regardless of whether the helpful hint it emitted or not +the causal 
exception must always be propagated+.

My vote is to back out FLINK-4663 and maybe add some logging instead.



> Don't swallow exception when rethrowing type mismatches with side outputs
> -
>
> Key: FLINK-14174
> URL: https://issues.apache.org/jira/browse/FLINK-14174
> Project: Flink
>  Issue Type: Improvement
>  Components: API / DataStream
>Affects Versions: 1.8.1, 1.9.0
>Reporter: John Lonergan
>Priority: Major
>
> The change made by https://github.com/apache/flink/pull/4663/files introduces 
> a "helpful hint" that a class cast at that location might be due a Flink 
> pipeline cfg error and the code goes onto swallows the original exception and 
> masked a protocol buffer serialisation problem we were having.
> This "helpful hint" is sometimes a complete red herring and in my case wasted 
> a lot of peoples time.
> In my case I had a class cast error in some proto serialisation code and 
> because the "helpful hint" traps ClassCastException I wasn't able to discover 
> the error easily. In the end we modified the Flink distribution to remove 
> this "helpful hint" at which point the real error was found and we quickly 
> fixed it - but not without a lot of burned time.
> I am not convinced of the 

[jira] [Updated] (FLINK-14174) Don't swallow exception when rethrowing type mismatches with side outputs

2019-09-23 Thread John Lonergan (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-14174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Lonergan updated FLINK-14174:
--
Description: 
The change made by https://github.com/apache/flink/pull/4663/files introduces a 
"helpful hint" that a class cast at that location might be due a Flink pipeline 
cfg error and the code goes onto swallows the original exception and masked a 
protocol buffer serialisation problem we were having.

This "helpful hint" is sometimes a complete red herring and in my case wasted a 
lot of peoples time.

In my case I had a class cast error in some proto serialisation code and 
because the "helpful hint" traps ClassCastException I wasn't able to discover 
the error easily. In the end we modified the Flink distribution to remove this 
"helpful hint" at which point the real error was found and we quickly fixed it 
- but not without a lot of burned time.

I am not convinced of the cost/benefit of the "helpful hint" introduced by 
FLINK-4663 for two reasons 
- it can be a red herring - in mine case and also and [at least one other 
person 
|https://stackoverflow.com/questions/56069797/multiple-outputtags-in-stream-process-function-with-classcastexception
]
- I don't agree with ever throwing away or masking causal exceptions - these 
must always be propagated (I raised a similar issue in my previous contribution)



My suggestion is to either back out FLINK-4663 so that we get to see the raw 
underlying exception and call stack or come up with a way to distinguish the 
specific case "FLINK-4663" was attempting to cover and only emit that hint hint 
if the specific case is encountered. 
For all other cases the helpful hint should not be emitted.  
And - regardless of whether the helpful hint it emitted or not +the causal 
exception must always be propagated+.

My vote is to back out FLINK-4663 and maybe add some logging instead.


  was:
The change made by https://github.com/apache/flink/pull/4663/files swallows the 
original exception.

Whilst I am in favour of adding additional helpful tips (which was the purpose 
of FLINK-4663) I don't agree with throwing away or masking causal exceptions.

IMHO the correct approach is to add the helpful hint as the first arg to "new 
ExceptionInChainedOperatorException(msg, ex)" and pass the original class cast 
ex as the cause.

Ie change this .. 
https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/OperatorChain.java#L672



> Don't swallow exception when rethrowing type mismatches with side outputs
> -
>
> Key: FLINK-14174
> URL: https://issues.apache.org/jira/browse/FLINK-14174
> Project: Flink
>  Issue Type: Improvement
>  Components: API / DataStream
>Affects Versions: 1.8.1, 1.9.0
>Reporter: John Lonergan
>Priority: Major
>
> The change made by https://github.com/apache/flink/pull/4663/files introduces 
> a "helpful hint" that a class cast at that location might be due a Flink 
> pipeline cfg error and the code goes onto swallows the original exception and 
> masked a protocol buffer serialisation problem we were having.
> This "helpful hint" is sometimes a complete red herring and in my case wasted 
> a lot of peoples time.
> In my case I had a class cast error in some proto serialisation code and 
> because the "helpful hint" traps ClassCastException I wasn't able to discover 
> the error easily. In the end we modified the Flink distribution to remove 
> this "helpful hint" at which point the real error was found and we quickly 
> fixed it - but not without a lot of burned time.
> I am not convinced of the cost/benefit of the "helpful hint" introduced by 
> FLINK-4663 for two reasons 
> - it can be a red herring - in mine case and also and [at least one other 
> person 
> |https://stackoverflow.com/questions/56069797/multiple-outputtags-in-stream-process-function-with-classcastexception
> ]
> - I don't agree with ever throwing away or masking causal exceptions - these 
> must always be propagated (I raised a similar issue in my previous 
> contribution)
> 
> My suggestion is to either back out FLINK-4663 so that we get to see the raw 
> underlying exception and call stack or come up with a way to distinguish the 
> specific case "FLINK-4663" was attempting to cover and only emit that hint 
> hint if the specific case is encountered. 
> For all other cases the helpful hint should not be emitted.  
> And - regardless of whether the helpful hint it emitted or not +the causal 
> exception must always be propagated+.
> My vote is to back out FLINK-4663 and maybe add some logging instead.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (FLINK-14174) Don't swallow exception when rethrowing type mismatches with side outputs

2019-09-23 Thread John Lonergan (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-14174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Lonergan updated FLINK-14174:
--
Description: 
The change made by https://github.com/apache/flink/pull/4663/files introduces a 
"helpful hint" that a class cast at that location might be due a Flink pipeline 
cfg error and the code goes onto swallows the original exception and masked a 
protocol buffer serialisation problem we were having.

Recorded as a bug because this "helpful hint" masks exceptions and is sometimes 
a complete red herring and in my case wasted a lot of peoples time.

In my case I had a class cast error in some proto serialisation code and 
because the "helpful hint" traps ClassCastException I wasn't able to discover 
the error easily. In the end we modified the Flink distribution to remove this 
"helpful hint" at which point the real error was found and we quickly fixed it 
- but not without a lot of burned time.

I am not convinced of the cost/benefit of the "helpful hint" introduced by 
FLINK-4663 for two reasons 
- it can be a red herring - in mine case and also and [at least one other 
person 
|https://stackoverflow.com/questions/56069797/multiple-outputtags-in-stream-process-function-with-classcastexception
]
- I don't agree with ever throwing away or masking causal exceptions - these 
must always be propagated (I raised a similar issue in my previous contribution)



My suggestion is to either back out FLINK-4663 so that we get to see the raw 
underlying exception and call stack -or- come up with a way to distinguish the 
specific case "FLINK-4663" was attempting to cover and only emit that hint hint 
if the specific case is encountered. 
For all other cases the helpful hint should not be emitted.  
And - regardless of whether the helpful hint it emitted or not +the causal 
exception must always be propagated+.

My vote is to back out FLINK-4663 and maybe add some logging instead.


  was:
The change made by https://github.com/apache/flink/pull/4663/files introduces a 
"helpful hint" that a class cast at that location might be due a Flink pipeline 
cfg error and the code goes onto swallows the original exception and masked a 
protocol buffer serialisation problem we were having.

This "helpful hint" is sometimes a complete red herring and in my case wasted a 
lot of peoples time.

In my case I had a class cast error in some proto serialisation code and 
because the "helpful hint" traps ClassCastException I wasn't able to discover 
the error easily. In the end we modified the Flink distribution to remove this 
"helpful hint" at which point the real error was found and we quickly fixed it 
- but not without a lot of burned time.

I am not convinced of the cost/benefit of the "helpful hint" introduced by 
FLINK-4663 for two reasons 
- it can be a red herring - in mine case and also and [at least one other 
person 
|https://stackoverflow.com/questions/56069797/multiple-outputtags-in-stream-process-function-with-classcastexception
]
- I don't agree with ever throwing away or masking causal exceptions - these 
must always be propagated (I raised a similar issue in my previous contribution)



My suggestion is to either back out FLINK-4663 so that we get to see the raw 
underlying exception and call stack -or- come up with a way to distinguish the 
specific case "FLINK-4663" was attempting to cover and only emit that hint hint 
if the specific case is encountered. 
For all other cases the helpful hint should not be emitted.  
And - regardless of whether the helpful hint it emitted or not +the causal 
exception must always be propagated+.

My vote is to back out FLINK-4663 and maybe add some logging instead.



> Don't swallow exception when rethrowing type mismatches with side outputs
> -
>
> Key: FLINK-14174
> URL: https://issues.apache.org/jira/browse/FLINK-14174
> Project: Flink
>  Issue Type: Bug
>  Components: API / DataStream
>Affects Versions: 1.8.1, 1.9.0
>Reporter: John Lonergan
>Priority: Major
>
> The change made by https://github.com/apache/flink/pull/4663/files introduces 
> a "helpful hint" that a class cast at that location might be due a Flink 
> pipeline cfg error and the code goes onto swallows the original exception and 
> masked a protocol buffer serialisation problem we were having.
> Recorded as a bug because this "helpful hint" masks exceptions and is 
> sometimes a complete red herring and in my case wasted a lot of peoples time.
> In my case I had a class cast error in some proto serialisation code and 
> because the "helpful hint" traps ClassCastException I wasn't able to discover 
> the error easily. In the end we modified the Flink distribution to remove 
> this "helpful hint" at which point the real error was found and we 

[jira] [Updated] (FLINK-14174) Don't swallow exception when rethrowing type mismatches with side outputs

2019-09-23 Thread John Lonergan (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-14174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Lonergan updated FLINK-14174:
--
Description: 
The change made by https://github.com/apache/flink/pull/4663/files introduces a 
"helpful hint" that a class cast at that location might be due a Flink pipeline 
cfg error and the code goes onto swallows the original exception and masked a 
protocol buffer serialisation problem we were having.

Recorded as a bug because this "helpful hint" masks exceptions and is sometimes 
a complete red herring and in my case wasted a lot of peoples time.

In my case I had a class cast error in some proto serialisation code and 
because the "helpful hint" traps ClassCastException I wasn't able to discover 
the error easily. In the end we modified the Flink distribution to remove this 
"helpful hint" at which point the real error was found and we quickly fixed it 
- but not without a lot of burned time.

I am not convinced of the cost/benefit of the "helpful hint" introduced by 
FLINK-4663 for two reasons 
- it can be a red herring - in mine case and also and [at least one other 
person 
|https://stackoverflow.com/questions/56069797/multiple-outputtags-in-stream-process-function-with-classcastexception]
- I don't agree with ever throwing away or masking causal exceptions - these 
must always be propagated (I raised a similar issue in my previous contribution)



My suggestion is to either back out FLINK-4663 so that we get to see the raw 
underlying exception and call stack -or- come up with a way to distinguish the 
specific case "FLINK-4663" was attempting to cover and only emit that hint hint 
if the specific case is encountered. 
For all other cases the helpful hint should not be emitted.  
And - regardless of whether the helpful hint it emitted or not +the causal 
exception must always be propagated+.

My vote is to back out FLINK-4663 and maybe add some logging instead.


  was:
The change made by https://github.com/apache/flink/pull/4663/files introduces a 
"helpful hint" that a class cast at that location might be due a Flink pipeline 
cfg error and the code goes onto swallows the original exception and masked a 
protocol buffer serialisation problem we were having.

Recorded as a bug because this "helpful hint" masks exceptions and is sometimes 
a complete red herring and in my case wasted a lot of peoples time.

In my case I had a class cast error in some proto serialisation code and 
because the "helpful hint" traps ClassCastException I wasn't able to discover 
the error easily. In the end we modified the Flink distribution to remove this 
"helpful hint" at which point the real error was found and we quickly fixed it 
- but not without a lot of burned time.

I am not convinced of the cost/benefit of the "helpful hint" introduced by 
FLINK-4663 for two reasons 
- it can be a red herring - in mine case and also and [at least one other 
person 
|https://stackoverflow.com/questions/56069797/multiple-outputtags-in-stream-process-function-with-classcastexception
]
- I don't agree with ever throwing away or masking causal exceptions - these 
must always be propagated (I raised a similar issue in my previous contribution)



My suggestion is to either back out FLINK-4663 so that we get to see the raw 
underlying exception and call stack -or- come up with a way to distinguish the 
specific case "FLINK-4663" was attempting to cover and only emit that hint hint 
if the specific case is encountered. 
For all other cases the helpful hint should not be emitted.  
And - regardless of whether the helpful hint it emitted or not +the causal 
exception must always be propagated+.

My vote is to back out FLINK-4663 and maybe add some logging instead.



> Don't swallow exception when rethrowing type mismatches with side outputs
> -
>
> Key: FLINK-14174
> URL: https://issues.apache.org/jira/browse/FLINK-14174
> Project: Flink
>  Issue Type: Bug
>  Components: API / DataStream
>Affects Versions: 1.8.1, 1.9.0
>Reporter: John Lonergan
>Priority: Major
>
> The change made by https://github.com/apache/flink/pull/4663/files introduces 
> a "helpful hint" that a class cast at that location might be due a Flink 
> pipeline cfg error and the code goes onto swallows the original exception and 
> masked a protocol buffer serialisation problem we were having.
> Recorded as a bug because this "helpful hint" masks exceptions and is 
> sometimes a complete red herring and in my case wasted a lot of peoples time.
> In my case I had a class cast error in some proto serialisation code and 
> because the "helpful hint" traps ClassCastException I wasn't able to discover 
> the error easily. In the end we modified the Flink distribution to remove 
> this "helpful hint" at 

[jira] [Commented] (FLINK-14170) Support hadoop < 2.7 with StreamingFileSink.BulkFormatBuilder

2019-10-17 Thread John Lonergan (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-14170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953637#comment-16953637
 ] 

John Lonergan commented on FLINK-14170:
---

Hi I disagree with that approach as it's impact time to marker and effort in 
fixing the bug significantly.

This existing impl is an attempt at "fail early" and is a *nice to have 
feature*, 
however this implementation needlessly disables the product on 2.6 and is 
therefore a *major bug* for us users.



Can we split the discussion into 
1 remove the bug (ie the block)
2 other nice to have improvements



Re 1 - 

When removing the bug the requestor's suggested an optional addition ie 
including a simple  NotImplementedException in the Flink code - this seems like 
a reasonable *but optional* compromise to improve the quality of the error 
message for any unfortunate's who go via a code path that attempts to use 
truncate() on 2.6. That approach is a practical solution that satisfies both 
the need to correctness and helpfulness without completely blocking the use of 
this product for a large group of potential users, particularly the many-many 
slower moving enterprises out here in the wild.

Let's not add additional barriers in the way of fixing the primary issue.

Re 2 - Your points ... 

"we should fail at build time" - how is the possible - we don't target 
specifically hadoop 2.6 or other versions?
" pre-flight time" - again how is this possible - I've looked at this and it's 
pretty hard to see how that would work - I can't see an straightforward one 
(suggest you make a proposal on how this would work)
"same strategy" - not needed to fix the bug - and this is separate problem that 
needs a separate ticket


Re "time bomb waiting to explode" - hardly an reasonable description of the 
issue - it's not like the first time I would run this code is in production? 
I'd discover the issue within an hour or so of writing my prototype or 
implementation - not a big deal IMHO. And not a big deal at all if the helpful 
error message that the original question suggests was included in the solution.

*Again can I stress that we separate the critical bug (this 2.6 check) from 
other nice to haves*



> Support hadoop < 2.7 with StreamingFileSink.BulkFormatBuilder
> -
>
> Key: FLINK-14170
> URL: https://issues.apache.org/jira/browse/FLINK-14170
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / FileSystem
>Affects Versions: 1.8.0, 1.8.1, 1.8.2, 1.9.0
>Reporter: Bhagavan
>Priority: Major
>
> Currently, StreamingFileSink is supported only with Hadoop >= 2.7 
> irrespective of Row/bulk format builder. This restriction is due to truncate 
> is not supported in  Hadoop < 2.7
> However, BulkFormatBuilder does not use truncate method to restore the file. 
> So the restricting StreamingFileSink.BulkFormatBuilder to be used only with 
> Hadoop >= 2.7 is not necessary.
> So requested improvement is to remove the precondition on 
> HadoopRecoverableWriter and allow  BulkFormatBuilder (Parquet) to be used in 
> Hadoop 2.6 ( Most of the enterprises still on CDH 5.x)
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Comment Edited] (FLINK-14170) Support hadoop < 2.7 with StreamingFileSink.BulkFormatBuilder

2019-10-17 Thread John Lonergan (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-14170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953637#comment-16953637
 ] 

John Lonergan edited comment on FLINK-14170 at 10/17/19 11:24 AM:
--

Hi I disagree with your approach Kostas as it impacts time to marker and adds a 
huge effort to fixing this critical bug.

This existing impl is an attempt at "fail early" and is a *nice to have 
feature*, 
however this implementation needlessly disables the product on 2.6 and is 
therefore a *major bug* for us users.



Can we split the discussion into 
1 remove the bug (ie the block)
2 other nice to have improvements



Re 1 - 

When removing the bug the requestor's suggested an optional addition ie 
including a simple  NotImplementedException in the Flink code - this seems like 
a reasonable *but optional* compromise to improve the quality of the error 
message for any unfortunate's who go via a code path that attempts to use 
truncate() on 2.6. That approach is a practical solution that satisfies both 
the need to correctness and helpfulness without completely blocking the use of 
this product for a large group of potential users, particularly the many-many 
slower moving enterprises out here in the wild.

Let's not add additional barriers in the way of fixing the primary issue.

Re 2 - Your points ... 

"we should fail at build time" - how is the possible - we don't target 
specifically hadoop 2.6 or other versions?
" pre-flight time" - again how is this possible - I've looked at this and it's 
pretty hard to see how that would work - I can't see an straightforward one 
(suggest you make a proposal on how this would work)
"same strategy" - not needed to fix the bug - and this is separate problem that 
needs a separate ticket


Re "time bomb waiting to explode" - hardly an reasonable description of the 
issue - it's not like the first time I would run this code is in production? 
I'd discover the issue within an hour or so of writing my prototype or 
implementation - not a big deal IMHO. And not a big deal at all if the helpful 
error message that the original question suggests was included in the solution.

*Again can I stress that we separate the critical bug (this 2.6 check) from 
other nice to haves*




was (Author: johnlon):
Hi I disagree with that approach as it's impact time to marker and effort in 
fixing the bug significantly.

This existing impl is an attempt at "fail early" and is a *nice to have 
feature*, 
however this implementation needlessly disables the product on 2.6 and is 
therefore a *major bug* for us users.



Can we split the discussion into 
1 remove the bug (ie the block)
2 other nice to have improvements



Re 1 - 

When removing the bug the requestor's suggested an optional addition ie 
including a simple  NotImplementedException in the Flink code - this seems like 
a reasonable *but optional* compromise to improve the quality of the error 
message for any unfortunate's who go via a code path that attempts to use 
truncate() on 2.6. That approach is a practical solution that satisfies both 
the need to correctness and helpfulness without completely blocking the use of 
this product for a large group of potential users, particularly the many-many 
slower moving enterprises out here in the wild.

Let's not add additional barriers in the way of fixing the primary issue.

Re 2 - Your points ... 

"we should fail at build time" - how is the possible - we don't target 
specifically hadoop 2.6 or other versions?
" pre-flight time" - again how is this possible - I've looked at this and it's 
pretty hard to see how that would work - I can't see an straightforward one 
(suggest you make a proposal on how this would work)
"same strategy" - not needed to fix the bug - and this is separate problem that 
needs a separate ticket


Re "time bomb waiting to explode" - hardly an reasonable description of the 
issue - it's not like the first time I would run this code is in production? 
I'd discover the issue within an hour or so of writing my prototype or 
implementation - not a big deal IMHO. And not a big deal at all if the helpful 
error message that the original question suggests was included in the solution.

*Again can I stress that we separate the critical bug (this 2.6 check) from 
other nice to haves*



> Support hadoop < 2.7 with StreamingFileSink.BulkFormatBuilder
> -
>
> Key: FLINK-14170
> URL: https://issues.apache.org/jira/browse/FLINK-14170
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / FileSystem
>Affects Versions: 1.8.0, 1.8.1, 1.8.2, 1.9.0
>Reporter: Bhagavan
>Priority: Major
>
> Currently, StreamingFileSink is supported only with Hadoop >= 2.7 
> irrespective of Row/bulk format builder. This 

[jira] [Comment Edited] (FLINK-14170) Support hadoop < 2.7 with StreamingFileSink.BulkFormatBuilder

2019-10-17 Thread John Lonergan (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-14170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953637#comment-16953637
 ] 

John Lonergan edited comment on FLINK-14170 at 10/17/19 11:25 AM:
--

Hi I disagree with your approach Kostas as it impacts time to marker and adds a 
huge effort to fixing this critical bug.

This existing impl is an attempt at "fail early" and is a *nice to have 
feature*, 
however this implementation needlessly disables the product on 2.6 and is 
therefore a *major bug* for us users.



Can we split the discussion into 
1 remove the bug (ie the block)
2 other nice to have improvements



Re 1 - 

When removing the bug the requestor's suggested an optional addition ie 
including a simple  NotImplementedException in the Flink code - this seems like 
a reasonable *but optional* compromise to improve the quality of the error 
message for any unfortunate's who go via a code path that attempts to use 
truncate() on 2.6. That approach is a practical solution that satisfies both 
the need to correctness and helpfulness without completely blocking the use of 
this product for a large group of potential users, particularly the many-many 
slower moving enterprises out here in the wild.

Let's not add additional barriers in the way of fixing the primary issue.

Re 2 - Your points ... 

"we should fail at build time" - how is the possible - we don't target 
specifically hadoop 2.6 or other versions?
" pre-flight time" - again how is this possible - I've looked at this and it's 
pretty hard to see how that would work - I can't see an straightforward one 
(suggest you make a proposal on how this would work)
"same strategy" - not needed to fix the bug - and this is separate problem that 
needs a separate ticket


Re "time bomb waiting to explode" - hardly an reasonable description of the 
issue - it's not like the first time I would run this code is in production? 
I'd discover the issue within an hour or so of writing my prototype or 
implementation - not a big deal IMHO. And not a big deal at all if the helpful 
error message that the original question suggests was included in the solution.

---

Again can I stress that we separate the critical bug (this 2.6 check) from 
other nice to haves




was (Author: johnlon):
Hi I disagree with your approach Kostas as it impacts time to marker and adds a 
huge effort to fixing this critical bug.

This existing impl is an attempt at "fail early" and is a *nice to have 
feature*, 
however this implementation needlessly disables the product on 2.6 and is 
therefore a *major bug* for us users.



Can we split the discussion into 
1 remove the bug (ie the block)
2 other nice to have improvements



Re 1 - 

When removing the bug the requestor's suggested an optional addition ie 
including a simple  NotImplementedException in the Flink code - this seems like 
a reasonable *but optional* compromise to improve the quality of the error 
message for any unfortunate's who go via a code path that attempts to use 
truncate() on 2.6. That approach is a practical solution that satisfies both 
the need to correctness and helpfulness without completely blocking the use of 
this product for a large group of potential users, particularly the many-many 
slower moving enterprises out here in the wild.

Let's not add additional barriers in the way of fixing the primary issue.

Re 2 - Your points ... 

"we should fail at build time" - how is the possible - we don't target 
specifically hadoop 2.6 or other versions?
" pre-flight time" - again how is this possible - I've looked at this and it's 
pretty hard to see how that would work - I can't see an straightforward one 
(suggest you make a proposal on how this would work)
"same strategy" - not needed to fix the bug - and this is separate problem that 
needs a separate ticket


Re "time bomb waiting to explode" - hardly an reasonable description of the 
issue - it's not like the first time I would run this code is in production? 
I'd discover the issue within an hour or so of writing my prototype or 
implementation - not a big deal IMHO. And not a big deal at all if the helpful 
error message that the original question suggests was included in the solution.

*Again can I stress that we separate the critical bug (this 2.6 check) from 
other nice to haves*



> Support hadoop < 2.7 with StreamingFileSink.BulkFormatBuilder
> -
>
> Key: FLINK-14170
> URL: https://issues.apache.org/jira/browse/FLINK-14170
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / FileSystem
>Affects Versions: 1.8.0, 1.8.1, 1.8.2, 1.9.0
>Reporter: Bhagavan
>Priority: Major
>
> Currently, StreamingFileSink is supported only with Hadoop >= 2.7 
> irrespective of Row/bulk format