Hi Nagarajan,
Do you see any exceptions in the logs ? Can you please give it a try to
ingest > 100 records and see if that works. Also, can you please share
the version of Phoenix you are using.
Regards
Ravi
On Thu, Dec 18, 2014 at 10:36 PM, Divya Nagarajan <[email protected]>
wrote:
>
>
> H i,
> I tried with 5 as batchsize,still data is not upserted into phoenix.
>
>
> 14/12/19 12:03:51 INFO zookeeper.ClientCnxn: EventThread shut down
> 14/12/19 12:03:51 INFO zookeeper.ZooKeeper: Session: 0x14a60b4631e003d
> closed
> 14/12/19 12:03:55 INFO serializer.BaseEventSerializer: the upsert
> statement is UPSERT INTO APACHE_LOG4 ("HOST", "IDENTITY", "USER", "TIME",
> "METHOD", "REQUEST", "PROTOCOL", "STATUS", "SIZE", "REFERER", "AGENT",
> "F_HOST", "UID") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
> 14/12/19 12:03:55 INFO sink.PhoenixSink: Time taken to process [3] events
> was [0] seconds
> 14/12/19 12:03:55 INFO sink.PhoenixSink: Time taken to process [3] events
> was [0] seconds
> 14/12/19 12:03:55 INFO sink.PhoenixSink: Time taken to process [3] events
> was [0] seconds
> 14/12/19 12:03:58 INFO sink.PhoenixSink: Time taken to process [1] events
> was [3] seconds
> 14/12/19 12:04:02 INFO sink.PhoenixSink: Time taken to process [0] events
> was [3] seconds
> 14/12/19 12
>
>
> Thanks
>
> Divya N
>
>
> On Fri, Dec 19, 2014 at 6:10 AM, Ravi Kiran <[email protected]>
> wrote:
>>
>>
>> Hi Nagarajan,
>>
>> Apparently, we do batches of 100 by default for each commit . You can
>> decrease that number if you would like to.
>> http://phoenix.apache.org/flume.html
>>
>> Regards
>> Ravi
>>
>> Hi,
>>>
>>> As we are working in flume to store apache log into hbase/phoenix
>>> using phoenix-flume jars.Using flume conf file we could create table
>>> in phoenix and see the events get processed.while querying through
>>> phoenix there is no record stored.please suggest us steps to process
>>> apache access log as source and store data in phoenix as sink.
>>>
>>> Hbase : 0.98.8
>>> Phoenix:4.2.1
>>>
>>> Flume Configuration:
>>> ===============
>>>
>>> agent.channels.memory-channel.type = memory
>>> agent.sources.tail-source.type = exec
>>> agent.sources.tail-source.command = tail -F /var/log/httpd/access_log
>>> agent.sources.tail-source.channels = memory-channel
>>>
>>> agent.channels.memory-channel.type=memory
>>> agent.channels.memory-channel.transactionCapacity=100
>>> agent.channels.memory-channel.byteCapacityBufferPercentage=20
>>>
>>>
>>> agent.channels = memory-channel
>>> agent.sources = tail-source
>>> agent.sinks = phoenix-sink
>>>
>>>
>>> agent.sinks.phoenix-sink.type=org.apache.phoenix.flume.sink.PhoenixSink
>>> agent.sinks.phoenix-sink.channel=memory-channel
>>> agent.sinks.phoenix-sink.batchSize=100
>>> agent.sinks.phoenix-sink.table=APACHE_LOGS
>>> agent.sinks.phoenix-sink.ddl=CREATE TABLE IF NOT EXISTS
>>> APACHE_LOGS(uid VARCHAR NOT NULL,host VARCHAR,identity VARCHAR,user
>>> VARCHAR,time VARCHAR,method VARCHAR,request VARCHAR,protocol
>>> VARCHAR,status INTEGER,size INTEGER,referer VARCHAR,agent
>>> VARCHAR,f_host VARCHAR CONSTRAINT pk PRIMARY KEY(uid))
>>> agent.sinks.phoenix-sink.zookeeperQuorum=dn02
>>> agent.sinks.phoenix-sink.serializer=REGEX
>>> agent.sinks.phoenix-sink.serializer.regex="^([\\d.]+) (\\S+) (\\S+)
>>> \\[([\\w:/]+\\s[+\\-]\\d{4})\\] \"(.+?)\" (\\d{3}) (\\d+) \"([^\"]+)\"
>>> \"([^\"]+)\""
>>> agent.sinks.phoenix-sink.serializer.rowkeyType=uuid
>>>
>>> agent.sinks.phoenix-sink.serializer.columns=host,identity,user,time,method,request,protocol,status,size,referer,agent
>>> agent.sinks.phoenix-sink.serializer.headers=f_host
>>>
>>> flume-ng agent -c conf -f /opt/flume/conf/phoenix.conf -n agent
>>>
>>> *
>>> *
>>> *
>>> *
>>> *
>>> *
>>>
>>> 14/12/18 12:53:09 INFO
>>> client.HConnectionManager$HConnectionImplementation: Closing master
>>> protocol: MasterService
>>> 14/12/18 12:53:09 INFO
>>> client.HConnectionManager$HConnectionImplementation: Closing zookeeper
>>> sessionid=0x14a5b888d18003e
>>> 14/12/18 12:53:09 INFO zookeeper.ZooKeeper: Session: 0x14a5b888d18003e
>>> closed
>>> 14/12/18 12:53:09 INFO zookeeper.ClientCnxn: EventThread shut down
>>> 14/12/18 12:53:09 INFO query.ConnectionQueryServicesImpl: Found
>>> quorum: nn01:2181
>>> 14/12/18 12:53:09 INFO zookeeper.RecoverableZooKeeper: Process
>>> identifier=hconnection-0x843735 connecting to ZooKeeper
>>> ensemble=nn01:2181
>>> 14/12/18 12:53:09 INFO zookeeper.ZooKeeper: Initiating client
>>> connection, connectString=nn01:2181 sessionTimeout=180000
>>> watcher=hconnection-0x843735, quorum=nn01:2181, baseZNode=/hbase
>>> 14/12/18 12:53:09 INFO zookeeper.ClientCnxn: Opening socket connection
>>> to server nn01/10.10.10.25:2181. Will not attempt to authenticate
>>> using SASL (unknown error)
>>> 14/12/18 12:53:09 INFO zookeeper.ClientCnxn: Socket connection
>>> established to nn01/10.10.10.25:2181, initiating session
>>> 14/12/18 12:53:09 INFO zookeeper.ClientCnxn: Session establishment
>>> complete on server nn01/10.10.10.25:2181, sessionid =
>>> 0x14a5b888d18003f, negotiated timeout = 40000
>>> 14/12/18 12:53:09 INFO
>>> client.HConnectionManager$HConnectionImplementation: Closing master
>>> protocol: MasterService
>>> 14/12/18 12:53:09 INFO
>>> client.HConnectionManager$HConnectionImplementation: Closing zookeeper
>>> sessionid=0x14a5b888d18003f
>>> 14/12/18 12:53:09 INFO zookeeper.ZooKeeper: Session: 0x14a5b888d18003f
>>> closed
>>> 14/12/18 12:53:09 INFO zookeeper.ClientCnxn: EventThread shut down
>>> 14/12/18 12:53:15 INFO serializer.BaseEventSerializer: the upsert
>>> statement is UPSERT INTO APACHE_LOG4 ("HOST", "IDENTITY", "USER",
>>> "TIME", "METHOD", "REQUEST", "PROTOCOL", "STATUS", "SIZE", "REFERER",
>>> "AGENT", "F_HOST", "UID") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
>>> ?)
>>> 14/12/18 12:53:18 INFO sink.PhoenixSink: Time taken to process [10]
>>> events was [3] seconds
>>> 14/12/18 12:53:22 INFO sink.PhoenixSink: Time taken to process [0]
>>> events was [3] seconds
>>> 14/12/18 12:53:27 INFO sink.PhoenixSink: Time taken to process [0]
>>> events was [3] seconds
>>> 14/12/18 12:53:33 INFO sink.PhoenixSink: Time taken to process [0]
>>> events was [3] seconds
>>> 14/12/18 12:53:40 INFO sink.PhoenixSink: Time taken to process [0]
>>> events was [3] seconds
>>> 14/12/18 12:53:48 INFO sink.PhoenixSink: Time taken to process [0]
>>> events was [3] seconds
>>> 14/12/18 12:53:56 INFO sink.PhoenixSink: Time taken to process [0]
>>> events was [3] seconds
>>> 14/12/18 12:54:04 INFO sink.PhoenixSink: Time taken to process [0]
>>> events was [3] seconds
>>> 14/12/18 12:54:12 INFO sink.PhoenixSink: Time taken to process [0]
>>> events was [3] seconds
>>> 14/12/18 12:54:20 INFO sink.PhoenixSink: Time taken to process [0]
>>> events was [3] seconds
>>> 14/12/18 12:54:28 INFO sink.PhoenixSink: Time taken to process [1]
>>> events was [3] seconds
>>> 14/12/18 12:54:36 INFO sink.PhoenixSink: Time taken to process [0]
>>> events was [3] seconds
>>>
>>>
>>> Thanks
>>>
>>> Divya N
>>>
>>
