Hi David, Todd. @David
>> *What do you mean that 2% are missing?* For example, after inserting 200M rows table has only 196M rows. >> *How are you checking that all the rows are there?* `SELECT COUNT(*) FROM tab` was executed via impala-shell. >> *do you still observe that rows are missing if you run the scans again?* Yes. 2 days past, but it has smaller rows. @Todd Here is the out of `kudu cluster ksck -checksum_scan -tables table_name`. Does it seem like having problems? ``` Table table_name is HEALTHY (440 tablet(s) checked) The metadata for 1 table(s) is HEALTHY Using snapshot timestamp: 6115681861904150528 Checksum running for 5s: 849/1320 replicas remaining (3.06G from disk, 19.89M rows summed) Checksum running for 10s: 849/1320 replicas remaining (6.76G from disk, 44.01M rows summed) ... ... Checksum finished in 352s: 0/1320 replicas remaining (100.37G from disk, 636.48M rows summed) ----------------------- table_name ----------------------- T 9ca22d9f67b6490986d3cd93ccfb3d58 P 380857bbccbd4bb3bddce021ffd1d89c (hostname:7050): Checksum: 0 T 9ca22d9f67b6490986d3cd93ccfb3d58 P 66b8e4e226844800aae13601458130b3 (hostname:7050): Checksum: 0 T 9ca22d9f67b6490986d3cd93ccfb3d58 P aec55b4e2ac140b6b57261db469427cf (hostname:7050): Checksum: 0 T 7d0b4aa457954529bfcbe2b9842424ea P 380857bbccbd4bb3bddce021ffd1d89c (hostname:7050): Checksum: 11616215297851188 T 7d0b4aa457954529bfcbe2b9842424ea P 66b8e4e226844800aae13601458130b3 (hostname:7050): Checksum: 11616215297851188 T 7d0b4aa457954529bfcbe2b9842424ea P a9764471770d43bdab279db074d4380f (hostname:7050): Checksum: 11616215297851188 ... T 770e22b6c0284523a15e688a86ab4f68 P 0abff808ff3e4248a9ddd97b01910e6c (hostname:7050): Checksum: 0 T 770e22b6c0284523a15e688a86ab4f68 P 0d1887e5d116477e82f655e3153afba4 (hostname:7050): Checksum: 0 T 770e22b6c0284523a15e688a86ab4f68 P 401b6963d32b42d79d5b5feda1de212b (hostname:7050): Checksum: 0 T 982615fa3b45461084dd6d60c3af9d4b P 7092c47bd9db4195887a521f17855b23 (hostname:7050): Checksum: 11126898289076092 T 982615fa3b45461084dd6d60c3af9d4b P 380857bbccbd4bb3bddce021ffd1d89c (hostname:7050): Checksum: 11126898289076092 T 982615fa3b45461084dd6d60c3af9d4b P a7ca07f9b3d944148b74c478bbb21cfb (hostname:7050): Checksum: 11126898289076092 ... ================== Errors: ================== error fetching info from tablet servers: Network error: Not all Tablet Servers are reachable FAILED Runtime error: ksck discovered errors ``` (One of tservers is still down, but it has no tablet of the invalid state table.) Regards, Jason 2017-04-25 6:45 GMT+09:00 Todd Lipcon <[email protected]>: > I think it's also worth trying 'kudu cluster ksck -checksum_scan > <master1,master2,master3>' to perform a consistency check. This will ensure > that the available replicas have matching data (and uses the SNAPSHOT scan > mode to avoid the inconsistency that David mentioned above). > > On Mon, Apr 24, 2017 at 2:38 PM, David Alves <[email protected]> > wrote: > >> Hi Jason >> >> What do you mean that 2% are missing? Were you not able to insert them >> (got a timeout) or where there no errors but you can't see the rows as the >> result of a scan? >> How are you checking that all the rows are there? Through a regular >> scan in spark? In particular the default ReadMode for scans makes no >> guarantees about replica recency, so it might happen that when you kill a >> tablet server, the other chosen replica is not up-to-date and returns less >> rows. In this case it's not that the rows are missing just that the replica >> that served the scan doesn't have them yet. >> These kinds of checks should likely be done with the READ_AT_SNAPSHOT >> ReadMode but even if you can't change ReadModes, do you still observe that >> rows are missing if you run the scans again? >> Currently some throttling might be required to make sure that the >> clients don't overload the server with writes which causes writes to start >> timing out. More efficient bulk loads is something we're working on right >> now. >> >> Best >> David >> >> >> On Sat, Apr 22, 2017 at 6:48 AM, Jason Heo <[email protected]> >> wrote: >> >>> Hi. >>> >>> I'm using Apache Kudu 1.2. I'm currently testing high availability of >>> Kudu. >>> >>> During bulk loading, one tserver is stopped via CDH Manager >>> intentionally and 2% of rows are missing. >>> >>> I use Spark 1.6 and package org.apache.kudu:kudu-spark_2.10:1.1.0 for >>> bulk loading. >>> >>> I got a error several times during insertion. Although 2% is lost when >>> tserver is stop and not started again, If I start it right after stopped, >>> there was no loss even though I got same error messages. >>> >>> >>> I watched Comcast's recent presentation at Strata Hadoop, They said that >>> >>> >>> Spark is recommended for large inserts to ensure handling failures >>>> >>>> >>> I'm curious Comcast has no issues with tserver failures and how can I >>> prevent rows from being lost. >>> >>> ---------------------------------- >>> >>> Below is an spark error message. ("01d....b64" is the killed one.) >>> >>> >>> java.lang.RuntimeException: failed to write 2 rows from DataFrame to >>> Kudu; sample errors: Timed out: RPC can not complete before timeout: >>> Batch{operations=2, tablet='1e83668a9fa44883897474eaa20a7cad' >>> [0x00000001323031362D3036, 0x00000001323031362D3037), >>> ignoreAllDuplicateRows=false, rpc=KuduRpc(method=Write, >>> tablet=1e83668a9fa44883897474eaa20a7cad, attempt=25, >>> DeadlineTracker(timeout=30000, elapsed=29298), Traces: [0ms] sending RPC to >>> server 01d513bc5c1847c29dd89c3d21a1eb64, [589ms] received from server >>> 01d513bc5c1847c29dd89c3d21a1eb64 response Network error: [Peer >>> 01d513bc5c1847c29dd89c3d21a1eb64] Connection reset, [589ms] delaying >>> RPC due to Network error: [Peer 01d513bc5c1847c29dd89c3d21a1eb64] >>> Connection reset, [597ms] querying master, [597ms] Sub rpc: >>> GetTableLocations sending RPC to server 50cb634c24ef426c9147cc4b7181ca11, >>> [599ms] Sub rpc: GetTableLocations sending RPC to server >>> 50cb634c24ef426c9147cc4b7181ca11, [643ms >>> ... >>> ... >>> received from server 01d513bc5c1847c29dd89c3d21a1eb64 response Network >>> error: [Peer 01d513bc5c1847c29dd89c3d21a1eb64] Connection reset, >>> [29357ms] delaying RPC due to Network error: [Peer >>> 01d513bc5c1847c29dd89c3d21a1eb64] Connection reset)} >>> at org.apache.kudu.spark.kudu.KuduContext$$anonfun$writeRows$1. >>> apply(KuduContext.scala:184) >>> at org.apache.kudu.spark.kudu.KuduContext$$anonfun$writeRows$1. >>> apply(KuduContext.scala:179) >>> at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfu >>> n$apply$33.apply(RDD.scala:920) >>> at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfu >>> n$apply$33.apply(RDD.scala:920) >>> at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkC >>> ontext.scala:1869) >>> at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkC >>> ontext.scala:1869) >>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) >>> at org.apache.spark.scheduler.Task.run(Task.scala:89) >>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) >>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >>> Executor.java:1142) >>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >>> lExecutor.java:617) >>> at java.lang.Thread.run(Thread.java:745) >>> ------------------ >>> >> >> > > > -- > Todd Lipcon > Software Engineer, Cloudera >
