[
https://issues.apache.org/jira/browse/FLINK-22436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Perez updated FLINK-22436:
--------------------------------
Description:
Sorry for selecting Connectors/Common, Twitter didn't show up in the Connectors
/ "X" list, I'm not sure why that is.
in addition to item FLINK-22435 I found this exception as well when the recent
version (1.12) twitter connector just hangs.
basically I am following the example here:
[https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/twitter/TwitterExample.java]
which just seems to run indefinitely but not have any data throughput/results
or exceptions.
task manager stdout (not logs) looks something like this:
{code:java}
WARNING: Please consider reporting this to the maintainers of
org.apache.flink.shaded.akka.org.jboss.netty.util.internal.ByteBufferUtilWARNING:
Use --illegal-access=warn to enable warnings of further illegal reflective
access operationsWARNING: All illegal access operations will be denied in a
future releaseMay 01, 2021 8:37:04 AM
org.apache.flink.twitter.shaded.com.google.common.io.Closeables closeWARNING:
IOException thrown while closing
Closeable.org.apache.http.MalformedChunkCodingException: CRLF expected at end
of chunk at
org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:253)
at
org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:225)
at
org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:184)
at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:213)
at
org.apache.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:315)
at
org.apache.http.conn.BasicManagedEntity.streamClosed(BasicManagedEntity.java:166)
at
org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228)
at
org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:172)
at
java.base/java.util.zip.InflaterInputStream.close(InflaterInputStream.java:231)
at java.base/java.util.zip.GZIPInputStream.close(GZIPInputStream.java:136)
at
org.apache.http.client.entity.LazyDecompressingInputStream.close(LazyDecompressingInputStream.java:94)
at
org.apache.flink.twitter.shaded.com.google.common.io.Closeables.close(Closeables.java:77)
at com.twitter.hbc.httpclient.Connection.close(Connection.java:64) at
com.twitter.hbc.httpclient.ClientBase.run(ClientBase.java:148) at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
May 01, 2021 8:39:06 AM
org.apache.flink.twitter.shaded.com.google.common.io.Closeables closeWARNING:
IOException thrown while closing
Closeable.org.apache.http.MalformedChunkCodingException: CRLF expected at end
of chunk at
org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:253)
at
org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:225)
at
org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:184)
at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:213)
at
org.apache.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:315)
at
org.apache.http.conn.BasicManagedEntity.streamClosed(BasicManagedEntity.java:166)
at
org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228)
at
org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:172)
at
java.base/java.util.zip.InflaterInputStream.close(InflaterInputStream.java:231)
at java.base/java.util.zip.GZIPInputStream.close(GZIPInputStream.java:136)
at
org.apache.http.client.entity.LazyDecompressingInputStream.close(LazyDecompressingInputStream.java:94)
at
org.apache.flink.twitter.shaded.com.google.common.io.Closeables.close(Closeables.java:77)
at com.twitter.hbc.httpclient.Connection.close(Connection.java:64) at
com.twitter.hbc.httpclient.ClientBase.run(ClientBase.java:148) at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
May 01, 2021 8:49:48 AM
org.apache.flink.twitter.shaded.com.google.common.io.Closeables closeWARNING:
IOException thrown while closing
Closeable.org.apache.http.MalformedChunkCodingException: CRLF expected at end
of chunk at
org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:253)
at
org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:225)
at
org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:184)
at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:213)
at
org.apache.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:315)
at
org.apache.http.conn.BasicManagedEntity.streamClosed(BasicManagedEntity.java:166)
at
org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228)
at
org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:172)
at
java.base/java.util.zip.InflaterInputStream.close(InflaterInputStream.java:231)
at java.base/java.util.zip.GZIPInputStream.close(GZIPInputStream.java:136)
at
org.apache.http.client.entity.LazyDecompressingInputStream.close(LazyDecompressingInputStream.java:94)
at
org.apache.flink.twitter.shaded.com.google.common.io.Closeables.close(Closeables.java:77)
at com.twitter.hbc.httpclient.Connection.close(Connection.java:64) at
com.twitter.hbc.httpclient.ClientBase.run(ClientBase.java:148) at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
{code}
and this just seems to repeat in the log.
I can provide more info if needed but my setup is really basic/simple.
was:
Sorry for selecting Connectors/Common, Twitter didn't show up in the Connectors
/ "X" list, I'm not sure why that is.
It appears the twitter datastream connector is getting an error previously
mentioned in a few other areas:
https://issues.apache.org/jira/browse/HTTPCLIENT-1763
[https://jira.spring.io/browse/SOCIAL-486]
basically I am following the example here:
https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/twitter/TwitterExample.java
which just seems to run indefinitely but not have any data throughput/results
or exceptions.
task manager log looks something like this:
{code:java}
2021-04-23 10:51:00,102 INFO
org.apache.flink.streaming.connectors.twitter.TwitterSource [] - Initializing
Twitter Streaming API connection2021-04-23 10:51:00,234 INFO
com.twitter.hbc.httpclient.BasicClient [] - New
connection executed: flink-twitter-source, endpoint:
/1.1/statuses/sample.json2021-04-23 10:51:00,234 INFO
org.apache.flink.streaming.connectors.twitter.TwitterSource [] - Twitter
Streaming API connection established successfully2021-04-23 10:51:00,254 INFO
com.twitter.hbc.httpclient.ClientBase [] -
flink-twitter-source Establishing a connection2021-04-23 10:51:00,861 WARN
org.apache.http.client.protocol.ResponseProcessCookies [] - Invalid
cookie header: "set-cookie: personalization_id="v1_eokcdF92+g1ZaDhF6ztjng==";
Max-Age=63072000; Expires=Sun, 23 Apr 2023 14:51:00 GMT; Path=/;
Domain=.twitter.com; Secure; SameSite=None". Invalid 'expires' attribute: Sun,
23 Apr 2023 14:51:00 GMT2021-04-23 10:51:00,861 WARN
org.apache.http.client.protocol.ResponseProcessCookies [] - Invalid
cookie header: "set-cookie: guest_id=v1%3A161918946086118384; Max-Age=63072000;
Expires=Sun, 23 Apr 2023 14:51:00 GMT; Path=/; Domain=.twitter.com; Secure;
SameSite=None". Invalid 'expires' attribute: Sun, 23 Apr 2023 14:51:00
GMT2021-04-23 10:51:00,865 INFO com.twitter.hbc.httpclient.ClientBase
[] - flink-twitter-source Processing connection data2021-04-23
10:51:40,448 INFO org.apache.flink.runtime.taskmanager.Task
[] - Attempting to cancel task Source: Custom Source -> Flat Map -> Sink:
Unnamed (1/1)#0 (360c4d19bf8443d3535c256f89f38215).2021-04-23 10:51:40,448 INFO
org.apache.flink.runtime.taskmanager.Task [] - Source:
Custom Source -> Flat Map -> Sink: Unnamed (1/1)#0
(360c4d19bf8443d3535c256f89f38215) switched from RUNNING to
CANCELING.2021-04-23 10:51:40,449 INFO
org.apache.flink.runtime.taskmanager.Task [] - Triggering
cancellation of task code Source: Custom Source -> Flat Map -> Sink: Unnamed
(1/1)#0 (360c4d19bf8443d3535c256f89f38215).2021-04-23 10:51:40,451 INFO
org.apache.flink.streaming.connectors.twitter.TwitterSource [] - Cancelling
Twitter source2021-04-23 10:51:40,452 INFO
org.apache.flink.streaming.connectors.twitter.TwitterSource [] - Closing
source2021-04-23 10:51:40,452 INFO com.twitter.hbc.httpclient.BasicClient
[] - Stopping the client: flink-twitter-source, endpoint:
/1.1/statuses/sample.json2021-04-23 10:51:40,452 INFO
com.twitter.hbc.httpclient.ClientBase [] -
flink-twitter-source exit event - Stopped by user: waiting for 5000
ms2021-04-23 10:51:40,547 INFO
org.apache.flink.streaming.connectors.twitter.TwitterSource [] - Closing
source2021-04-23 10:51:40,548 INFO com.twitter.hbc.httpclient.BasicClient
[] - Stopping the client: flink-twitter-source, endpoint:
/1.1/statuses/sample.json2021-04-23 10:51:40,585 INFO
com.twitter.hbc.httpclient.ClientBase [] -
flink-twitter-source Done processing, preparing to close connection2021-04-23
10:51:40,595 INFO com.twitter.hbc.httpclient.ClientBase
[] - flink-twitter-source Shutting down httpclient connection manager2021-04-23
10:51:40,596 INFO com.twitter.hbc.httpclient.BasicClient
[] - Successfully stopped the client: flink-twitter-source, endpoint:
/1.1/statuses/sample.json2021-04-23 10:51:40,596 INFO
com.twitter.hbc.httpclient.BasicClient [] - Successfully
stopped the client: flink-twitter-source, endpoint:
/1.1/statuses/sample.json2021-04-23 10:51:40,596 INFO
org.apache.flink.runtime.taskmanager.Task [] - Source:
Custom Source -> Flat Map -> Sink: Unnamed (1/1)#0
(360c4d19bf8443d3535c256f89f38215) switched from CANCELING to CANCELED.
{code}
and this just seems to repeat in the log.
I can provide more info if needed but my setup is really basic/simple.
> twitter datastream connector hangs
> ----------------------------------
>
> Key: FLINK-22436
> URL: https://issues.apache.org/jira/browse/FLINK-22436
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Connectors / Common
> Environment: My environment is just simple local flink env, using the
> twitter connector example.
> Reporter: Jason Perez
> Priority: Major
>
> Sorry for selecting Connectors/Common, Twitter didn't show up in the
> Connectors / "X" list, I'm not sure why that is.
>
> in addition to item FLINK-22435 I found this exception as well when the
> recent version (1.12) twitter connector just hangs.
>
> basically I am following the example here:
> [https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/twitter/TwitterExample.java]
>
> which just seems to run indefinitely but not have any data throughput/results
> or exceptions.
>
> task manager stdout (not logs) looks something like this:
>
> {code:java}
> WARNING: Please consider reporting this to the maintainers of
> org.apache.flink.shaded.akka.org.jboss.netty.util.internal.ByteBufferUtilWARNING:
> Use --illegal-access=warn to enable warnings of further illegal reflective
> access operationsWARNING: All illegal access operations will be denied in a
> future releaseMay 01, 2021 8:37:04 AM
> org.apache.flink.twitter.shaded.com.google.common.io.Closeables closeWARNING:
> IOException thrown while closing
> Closeable.org.apache.http.MalformedChunkCodingException: CRLF expected at end
> of chunk at
> org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:253)
> at
> org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:225)
> at
> org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:184)
> at
> org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:213)
> at
> org.apache.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:315)
> at
> org.apache.http.conn.BasicManagedEntity.streamClosed(BasicManagedEntity.java:166)
> at
> org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228)
> at
> org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:172)
> at
> java.base/java.util.zip.InflaterInputStream.close(InflaterInputStream.java:231)
> at
> java.base/java.util.zip.GZIPInputStream.close(GZIPInputStream.java:136) at
> org.apache.http.client.entity.LazyDecompressingInputStream.close(LazyDecompressingInputStream.java:94)
> at
> org.apache.flink.twitter.shaded.com.google.common.io.Closeables.close(Closeables.java:77)
> at com.twitter.hbc.httpclient.Connection.close(Connection.java:64) at
> com.twitter.hbc.httpclient.ClientBase.run(ClientBase.java:148) at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
> at java.base/java.lang.Thread.run(Thread.java:832)
> May 01, 2021 8:39:06 AM
> org.apache.flink.twitter.shaded.com.google.common.io.Closeables closeWARNING:
> IOException thrown while closing
> Closeable.org.apache.http.MalformedChunkCodingException: CRLF expected at end
> of chunk at
> org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:253)
> at
> org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:225)
> at
> org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:184)
> at
> org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:213)
> at
> org.apache.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:315)
> at
> org.apache.http.conn.BasicManagedEntity.streamClosed(BasicManagedEntity.java:166)
> at
> org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228)
> at
> org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:172)
> at
> java.base/java.util.zip.InflaterInputStream.close(InflaterInputStream.java:231)
> at
> java.base/java.util.zip.GZIPInputStream.close(GZIPInputStream.java:136) at
> org.apache.http.client.entity.LazyDecompressingInputStream.close(LazyDecompressingInputStream.java:94)
> at
> org.apache.flink.twitter.shaded.com.google.common.io.Closeables.close(Closeables.java:77)
> at com.twitter.hbc.httpclient.Connection.close(Connection.java:64) at
> com.twitter.hbc.httpclient.ClientBase.run(ClientBase.java:148) at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
> at java.base/java.lang.Thread.run(Thread.java:832)
> May 01, 2021 8:49:48 AM
> org.apache.flink.twitter.shaded.com.google.common.io.Closeables closeWARNING:
> IOException thrown while closing
> Closeable.org.apache.http.MalformedChunkCodingException: CRLF expected at end
> of chunk at
> org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:253)
> at
> org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:225)
> at
> org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:184)
> at
> org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:213)
> at
> org.apache.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:315)
> at
> org.apache.http.conn.BasicManagedEntity.streamClosed(BasicManagedEntity.java:166)
> at
> org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228)
> at
> org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:172)
> at
> java.base/java.util.zip.InflaterInputStream.close(InflaterInputStream.java:231)
> at
> java.base/java.util.zip.GZIPInputStream.close(GZIPInputStream.java:136) at
> org.apache.http.client.entity.LazyDecompressingInputStream.close(LazyDecompressingInputStream.java:94)
> at
> org.apache.flink.twitter.shaded.com.google.common.io.Closeables.close(Closeables.java:77)
> at com.twitter.hbc.httpclient.Connection.close(Connection.java:64) at
> com.twitter.hbc.httpclient.ClientBase.run(ClientBase.java:148) at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
> at java.base/java.lang.Thread.run(Thread.java:832)
> {code}
> and this just seems to repeat in the log.
> I can provide more info if needed but my setup is really basic/simple.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
