Re: Request for invitation to Slack team

2018-01-19 Thread Mingmin Xu 
Just sent, welcome!

On Fri, Jan 19, 2018 at 10:29 AM, Andrew Nguonly 
wrote:

> Request for invitation to Slack team
>



-- 

Mingmin

Re: Beam Slack Channel Invitation Request

2017-11-04 Thread Mingmin Xu 
sent, welcome!

On Sat, Nov 4, 2017 at 4:42 PM, Tristan Shephard  wrote:

> Hello,
>
> Can someone please add me to the Beam slack channel?
>
> Thanks in advance,
> Tristan
>



-- 

Mingmin

Re: Request to add to Beam Slack Channel

2017-11-04 Thread Mingmin Xu 
sent, welcome to Beam.

On Sat, Nov 4, 2017 at 7:44 PM, Ananth G  wrote:

> Hello,
>
> Could someone please add me to the Beam slack channel?
>
> Regards,
> Ananth
>



-- 

Mingmin

Re: Slack channel

2017-08-13 Thread Mingmin Xu 
sent, welcome Andy!

On Sat, Aug 12, 2017 at 10:59 PM, Andy Barron  wrote:

> Hi,
>
> I'd like to join the Slack channel for Apache Beam. I work at Maestro with
> Steve (CC'd), who was recently added (st...@maestro.io). My email is
> a...@maestro.io.
>
> Thanks!
> Andy
>



-- 

Mingmin

Re: slack invite

2017-08-07 Thread Mingmin Xu 
sent, welcome!

On Mon, Aug 7, 2017 at 12:44 PM,  wrote:

> hi there,
>
> could i get an invite to the apache beam slack channel please?
>
> thanks!
>
> - Reece
>
>
>
> Get Outlook for iOS 
>



-- 

Mingmin

Re: Slack invite

2017-08-07 Thread Mingmin Xu 
sent, welcome!

On Mon, Aug 7, 2017 at 2:20 PM, Akagi Norio 
wrote:

> Could I get a slack invitation?
> Thank you!
>



-- 

Mingmin

Re: Slack invite

2017-07-26 Thread Mingmin Xu 
done

On Wed, Jul 26, 2017 at 10:52 AM, Punit Naik <naik.puni...@gmail.com> wrote:

> Could I get one slack invite too, please?
>
>
> On Jul 26, 2017 11:20 PM, "Mingmin Xu" <mingm...@gmail.com> wrote:
>
> sent, welcome @Nathan.
>
> On Wed, Jul 26, 2017 at 10:47 AM, Nathan Deren <
> nathan.de...@zonarsystems.com> wrote:
>
>> Hi,
>>
>> Could I get a slack invite, please?
>>
>> Thanks very much!
>> —Nathan Deren
>>
>
>
>
> --
> 
> Mingmin
>
>
>


-- 

Mingmin

Re: Slack invite

2017-07-26 Thread Mingmin Xu 
sent, welcome @Nathan.

On Wed, Jul 26, 2017 at 10:47 AM, Nathan Deren <
nathan.de...@zonarsystems.com> wrote:

> Hi,
>
> Could I get a slack invite, please?
>
> Thanks very much!
> —Nathan Deren
>



-- 

Mingmin

Re: SQL in Stream Computing: MERGE or INSERT?

2017-06-22 Thread Mingmin Xu 
Would like to share my thoughts in another perspective. IMO this is a
typical scenario for column based databases, like Hbase/Cassandra. You may
need to choose a right database if possible.

UPSERT is another alternative option, but I wouldn't suggest to a
customized check-insert/check-update implementation. The actual job should
be done in database side.

On Thu, Jun 22, 2017 at 6:59 PM, James  wrote:

> Hi Tyler,
>
> I think upsert is a good alternative, concise as INSERT and have the valid
> semantics. Just that user seems rarely use UPSERT either(might because
> there's no UPDATE in batch big data processing).
>
> By *"INSERT will behave differently in batch & stream processing"* I
> mean, if we use the "INSERT" solution I described above, there will be ten
> INSERTs:
>
> *INSERT INTO result(rowkey, col1) values(...)*
>
> *INSERT INTO result(rowkey, col2) values(...)*
>
> *...INSERT INTO result(rowkey, col10) values(...)*
>
> Although we issued ten INSERTs, but there will be only ONE new records in
> the target table, because 9 of the INSERTs are actually UPDATing the
> record, so in stream computing *INSERT = (INSERT or UPDATE)*, while in
> batch,* INSERT is just INSERT*.
>
> I think the essence of this problem is, there is no UPDATE in batch, but
> require UPDATE in streaming.
>
>
>
> Tyler Akidau 于2017年6月22日周四 下午11:35写道：
>
>> Calcite appears to have UPSERT
>>  support, can we just
>> use that instead?
>>
>> Also, I don't understand your statement that "INSERT will behave
>> differently in batch & stream processing". Can you explain further?
>>
>>
>> -Tyler
>>
>>
>> On Thu, Jun 22, 2017 at 7:35 AM Jesse Anderson 
>> wrote:
>>
>>> If I'm understanding correctly, Hive does that with a insert into
>>> followed by a select statement that does the aggregation. https://cwiki.
>>> apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-
>>> InsertingdataintoHiveTablesfromqueries
>>>
>>> On Thu, Jun 22, 2017 at 1:32 AM James  wrote:
>>>
 Hi team,

 I am thinking about a SQL and stream computing related problem, want to
 hear your opinions.

 In stream computing, there is a typical case like this:

 *We want to calculate a big wide result table, which has one rowkey and
 ten
 value columns:*
 *create table result (*
 *rowkey varchar(127) PRIMARY KEY,*
 *col1 int,*
 *col2 int,*
 *...*
 *col10 int*
 *);*

 Each of the value columns is calculated by a complex query, so there
 will
 be ten SQLs to calculate
 data for this table, for each sql:

 * First check whether there is a row for the specified `rowkey`.
 * If yes, then `update`, otherwise `insert`.

 There is actually a dedicated sql syntax called `MERGE` designed for
 this(SQL2008), a sample usage is:

 MERGE INTO result D
USING (SELECT rowkey, col1 FROM input WHERE flag = 80) S
ON (D.rowkey = S.rowkey)
WHEN MATCHED THEN UPDATE SET D.col1 = S.col1
WHEN NOT MATCHED THEN INSERT (D.rowkey, D.col1)


 *The semantic fits perfectly, but it is very verbose, and normal users
 rarely used this syntax.*

 So my colleagues invented a new syntax for this scenario (Or more
 precisely, a new interpretation for the INSERT statement). For the above
 scenario, user will always write `insert` statement:

 insert into result(rowkey, col1) values(...)
 insert into result(rowkey, col2) values(...)

 The sql interpreter will do a trick behind the scene: if the `rowkey`
 exists, then update, otherwise `insert`. This solution is very concise,
 but
 violates the semantics of `insert`, using this solution INSERT will
 behave
 differently in batch & stream processing.

 How do you guys think? which do you prefer? What's your reasoning?

 Looking forward to your opinions, thanks in advance.

>>> --
>>> Thanks,
>>>
>>> Jesse
>>>
>>


-- 

Mingmin

Re: KafkaIO nothing received?

2017-05-04 Thread Mingmin Xu 
@Conrad,

Your code should be good to go, I can run it in my local env. There're two
points you may have a check:
1). does the topic have data there, you can confirm with kafka cli '
*bin/kafka-console-consumer.sh*';
2). is the port in bootstrapServers right? By default it's 9092.



On Thu, May 4, 2017 at 9:05 AM, Conrad Crampton  wrote:

> Hi,
>
> New to the group – ‘hello’!
>
>
>
> Just starting to look into Beam and I very much like the concepts, but
> have rather fallen at the first hurdle – that being trying to subscribe to
> a kafka topic and process results.
>
> Very simply the following code doesn’t get receive any records (the data
> is going into the queue) – I just get nothing.
>
> I have tried on both direct-runner and flink-runner (using the Quickstart
> as a base for options, mvn profile etc.)
>
>
>
> Code
>
>
>
> Pipeline p = Pipeline.*create*(options);
>
> List topics = ImmutableList.*of*(*"test-http-logs-json"*);
>
>
> PCollection logs = p.apply(KafkaIO.*read*()
> .withBootstrapServers(
> *"datanode2-cm1.mis-cds.local:6667,datanode3-cm1.mis-cds.local:6667,datanode6-cm1.mis-cds.local:6667"*
> )
> .withTopics(topics)
> .withKeyCoder(StringUtf8Coder.*of*())
> .withValueCoder(StringUtf8Coder.*of*())
> .withMaxNumRecords(10)
> .updateConsumerProperties(ImmutableMap.*builder*()
> .put(*"auto.offset.reset"*, (Object) *"earliest"*)
> .put(*"group.id "*, (Object)
> *"http-logs-beam-json"*)
> .put(*"enable.auto.commit"*, (Object) *"true"*)
> .put(*"receive.buffer.bytes"*, 1024 * 1024)
> .build())
>
> *// set a Coder for Key and Value *.withoutMetadata())
> .apply(*"Transform "*, MapElements.*via*(*new 
> *SimpleFunction String>, String>() {
> @Override
> *public *String apply(KV input) {
> *log*.debug(*"{}"*, input.getValue());
> *return *input.getKey() + *" " *+ input.getValue();
> }
> }));
>
>
> p.run();
>
>
>
>
>
> Result:
>
> May 04, 2017 5:02:13 PM org.apache.kafka.common.config.AbstractConfig
> logAll
>
> INFO: ConsumerConfig values:
>
> metric.reporters = []
>
> metadata.max.age.ms = 30
>
> value.deserializer = class org.apache.kafka.common.serialization.
> ByteArrayDeserializer
>
> group.id = http-logs-beam-json
>
> partition.assignment.strategy = [org.apache.kafka.clients.
> consumer.RangeAssignor]
>
> reconnect.backoff.ms = 50
>
> sasl.kerberos.ticket.renew.window.factor = 0.8
>
> max.partition.fetch.bytes = 1048576
>
> bootstrap.servers = [datanode2-cm1.mis-cds.local:6667,
> datanode3-cm1.mis-cds.local:6667, datanode6-cm1.mis-cds.local:6667]
>
> retry.backoff.ms = 100
>
> sasl.kerberos.kinit.cmd = /usr/bin/kinit
>
> sasl.kerberos.service.name = null
>
> sasl.kerberos.ticket.renew.jitter = 0.05
>
> ssl.keystore.type = JKS
>
> ssl.trustmanager.algorithm = PKIX
>
> enable.auto.commit = true
>
> ssl.key.password = null
>
> fetch.max.wait.ms = 500
>
> sasl.kerberos.min.time.before.relogin = 6
>
> connections.max.idle.ms = 54
>
> ssl.truststore.password = null
>
> session.timeout.ms = 3
>
> metrics.num.samples = 2
>
> client.id =
>
> ssl.endpoint.identification.algorithm = null
>
> key.deserializer = class org.apache.kafka.common.serialization.
> ByteArrayDeserializer
>
> ssl.protocol = TLS
>
> check.crcs = true
>
> request.timeout.ms = 4
>
>ssl.provider = null
>
> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
>
> ssl.keystore.location = null
>
> heartbeat.interval.ms = 3000
>
> auto.commit.interval.ms = 5000
>
> receive.buffer.bytes = 1048576
>
> ssl.cipher.suites = null
>
> ssl.truststore.type = JKS
>
> security.protocol = PLAINTEXT
>
> ssl.truststore.location = null
>
> ssl.keystore.password = null
>
> ssl.keymanager.algorithm = SunX509
>
> metrics.sample.window.ms = 3
>
> fetch.min.bytes = 1
>
> send.buffer.bytes = 131072
>
> auto.offset.reset = earliest
>
>
>
> May 04, 2017 5:02:13 PM org.apache.kafka.common.utils.AppInfoParser$AppInfo
> 
>
> INFO: Kafka version : 0.9.0.1
>
> May 04, 2017 5:02:13 PM org.apache.kafka.common.utils.AppInfoParser$AppInfo
> 
>
> INFO: Kafka commitId : 23c69d62a0cabf06
>
> May 04, 2017 5:02:13 PM 
> org.apache.beam.sdk.io.kafka.KafkaIO$UnboundedKafkaSource
> generateInitialSplits
>
> INFO: Partitions assigned to split 0 (total 1): test-http-logs-json-0
>
> May 04, 2017 5:02:13 PM org.apache.kafka.common.config.AbstractConfig
> logAll
>
> INFO: ConsumerConfig

Re: Slack Channel Request

2017-04-13 Thread Mingmin Xu 
both sent.

On Thu, Apr 13, 2017 at 10:00 PM, Anant Bhandarkar <
anant.bhandar...@impactanalytics.co> wrote:

> Would love to be part of beam group on slack.
> Also please add anil.b...@impactanaytics.co
> Thanks,
> Anant
>
> On 14-Apr-2017 9:15 AM, "Mingmin Xu" <mingm...@gmail.com> wrote:
>
>> @James, @Jingsong, @Tom, invite sent.
>>
>> On Thu, Apr 13, 2017 at 8:34 PM, Tom Pollard <
>> tpoll...@flashpoint-intel.com> wrote:
>>
>>> If it's not inconvenient, I'd also like an invitation to the Slack
>>> channel.
>>>
>>> Tom
>>>
>>>
>>> On Apr 13, 2017, at 11:31 PM, JingsongLee <lzljs3620...@aliyun.com>
>>> wrote:
>>>
>>> Please add me too.
>>>
>>> Best,
>>>
>>> JingsongLee
>>>
>>>
>>> --
>>> From:James <xumingmi...@gmail.com>
>>> Time:2017 Apr 14 (Fri) 11:00
>>> To:user <user@beam.apache.org>
>>> Subject:Re: Slack Channel Request
>>>
>>> Could I also have an invite please?
>>>
>>> On 2017-03-28 08:28 (+0800), Davor Bonaci <da...@apache.org> wrote:
>>> > Invite sent.
>>> >
>>> > On Sat, Mar 25, 2017 at 2:48 AM, Prabeesh K. <prabsma...@gmail.com
>>> > wrote:
>>> >
>>> > > Hi Jean,
>>> > >
>>> > > Thank you for your reply. I am eagerly waiting for the o
>>> ther options.
>>> > >
>>> > > Regards,
>>> > > Prabeesh K.
>>> > >
>>> > > On 25 March 2017 at 10:08, Jean-Baptiste Onofré <j...@nanthrax.net
>>> > wrote:
>>> > >
>>> > >> Unfortunately we reached the max number of people on Slack (90).
>>> > >>
>>> > >> Let me see what we can do.
>>> > >>
>>> > >> Regards
>>> > >> JB
>>> > >>
>>> > >>
>>> > >> On 03/24/2017 09:49 PM, Prabeesh K. wrote:
>>> > >>
>>> > >>> Hi,
>>> > >>>
>>> > >>> Can someone please add me to the Apache Beam slack channel?
>>> > >>>
>>> > >>> Regards,
>>> > >>>
>>> > >>> Prabeesh K.
>>> > >>>
>>> > >>>
>>> > >> --
>>> > >> Jean-Baptiste Onofré
>>> > >> jbono...@apache.org
>>> > >> http://blog.nanthrax.net
>>> > >> Talend - http://www.talend.com
>>> > >>
>>> > >
>>> > >
>>> >
>>>
>>>
>>>
>>
>>
>> --
>> 
>> Mingmin
>>
>


-- 

Mingmin

Re: Slack Channel Request

2017-04-13 Thread Mingmin Xu 
@James, @Jingsong, @Tom, invite sent.

On Thu, Apr 13, 2017 at 8:34 PM, Tom Pollard 
wrote:

> If it's not inconvenient, I'd also like an invitation to the Slack channel.
>
> Tom
>
>
> On Apr 13, 2017, at 11:31 PM, JingsongLee  wrote:
>
> Please add me too.
>
> Best,
>
> JingsongLee
>
>
> --
> From:James 
> Time:2017 Apr 14 (Fri) 11:00
> To:user 
> Subject:Re: Slack Channel Request
>
> Could I also have an invite please?
>
> On 2017-03-28 08:28 (+0800), Davor Bonaci  wrote:
> > Invite sent.
> >
> > On Sat, Mar 25, 2017 at 2:48 AM, Prabeesh K.  > wrote:
> >
> > > Hi Jean,
> > >
> > > Thank you for your reply. I am eagerly waiting for the other options.
> > >
> > > Regards,
> > > Prabeesh K.
> > >
> > > On 25 March 2017 at 10:08, Jean-Baptiste Onofré  > wrote:
> > >
> > >> Unfortunately we reached the max number of people on Slack (90).
> > >>
> > >> Let me see what we can do.
> > >>
> > >> Regards
> > >> JB
> > >>
> > >>
> > >> On 03/24/2017 09:49 PM, Prabeesh K. wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> Can someone please add me to the Apache Beam slack channel?
> > >>>
> > >>> Regards,
> > >>>
> > >>> Prabeesh K.
> > >>>
> > >>>
> > >> --
> > >> Jean-Baptiste Onofré
> > >> jbono...@apache.org
> > >> http://blog.nanthrax.net
> > >> Talend - http://www.talend.com
> > >>
> > >
> > >
> >
>
>
>


-- 

Mingmin

Re: Slack

2017-03-13 Thread Mingmin Xu 
added

ps, resent as it seems the previous one is blocked,

On Mon, Mar 13, 2017 at 1:07 PM, Sunil K Sahu  wrote:

> Could someone add me to slack channel as well.
>
> Thanks,
> Sunil
>
> ​
> Sunil Kumar Sahu
> ​CS
>  Dept - Graduate Student
> ​BU - ​Watson School of Engineering
>
> On Mon, Mar 13, 2017 at 9:28 AM, Amit Sela  wrote:
>
>> I'm so well trained, I do it on my phone now!
>>
>> On Mon, Mar 13, 2017, 15:24 Tobias Feldhaus <
>> tobias.feldh...@localsearch.ch> wrote:
>>
>>> Same for me please :)
>>>
>>> Tobi
>>>
>>>
>>>
>>> On 13.03.17, 13:30, "Amit Sela"  wrote:
>>>
>>>
>>>
>>> Done. Welcome!
>>>
>>>
>>>
>>> On Mon, Mar 13, 2017 at 2:29 PM Alexander Gallego <
>>> gallego.al...@gmail.com> wrote:
>>>
>>> same for me please.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> .alex
>>>
>>>
>>>
>>> On Fri, Mar 10, 2017 at 3:01 PM, Amit Sela  wrote:
>>>
>>> Done
>>>
>>>
>>>
>>> On Fri, Mar 10, 2017, 21:59 Devon Meunier 
>>> wrote:
>>>
>>> Hi!
>>>
>>>
>>>
>>> Sorry for the noise but could someone invite me to the slack channel?
>>>
>>>
>>>
>>> Thanks,
>>>
>>>
>>>
>>> Devon
>>>
>>>
>>>
>>>
>


-- 

Mingmin